The café cult of Patrick Downey

An ode to The Victory’s rugged but caring main character.
Photo of Patrick Downey, right, behind the counter, looking at a medium-sized dog with it's paws on the other side of the counter, waiting expectantly.
The Victory cafe owner Patrick Downey tosses a piece of ham to a dog over the counter on April 28, 2023. Photo by Alyssa Allemand

An ode to The Victory’s rugged but caring main character.

This story was produced in collaboration with Wisconsin Life. A co-production of Wisconsin Public Radio and PBS Wisconsin, Wisconsin Life celebrates the unique and diverse people, places, history and culture of the state. Read and listen to Wisconsin Life’s version of this story, and subscribe to the show wherever you get your podcasts.

Without saying a word to either the dog or the customer who’ve just walked through the doorway, The Victory owner Patrick Downey tosses pieces of ham and cheese for the pup to catch in its mouth from the other side of the counter. 

It’s a routine I’ve witnessed at the east-side coffee shop, simply labeled “CAFE” on a sign outside, a handful of times. (There’s no obvious sign on the building declaring “Victory café”—though there are articles about the café printed out on display lining the window.)

And I’ve noticed many of The Victory’s patrons have their own routines, all tucked in Downey’s memory. 

He makes a point to familiarize himself with his customers, tracking first names and go-to orders, complimenting outfits and hairstyles, finding commonalities in music and books and art—all while running the counter solo most days.

“If you come through the door of my shop, I have an almost pathological need to make you happy,” Downey says. “Like, it’s really the kind of fake niceness [that] just feels really awful. It just feels like hostility, fake niceness. So we don’t do that here. We don’t do passive here.” 

Downey opened the Madison location, at 2710 Atwood Ave., in August of 2010 after he and his wife moved to the city in 2009 to raise their family. But the café has a long history, originating in Brooklyn in 1999.

“I never had, like, a mission statement that was really geared towards doing much more than make rent,” Downey says. “That was sort of my mission statement when I opened in Brooklyn, was that I thought it would be good to make rent and not have to work for someone else.”

“The reason I opened a coffee shop is that I needed a job I couldn’t quit,” he continues. “I was almost 40 and I thought, ‘It’s cute when you’re quitting all these jobs, but when you’re 40, it’s just like, what’s wrong with this guy?’ So I got a job I couldn’t quit and I haven’t. Even though the early days in Brooklyn were, like, rough. Really, really, really, really rough.”

Downey’s the meeting of rugged, Irish-Catholic Brooklyn and an endearing Italian grandma, as one customer puts it. Upon anyone’s arrival, his first question is always, “Would you like a glass of water?” But he’s also not shy about enforcing the café’s no-phone rule. Customers aren’t allowed to take phone calls inside, and Downey has kicked people out for doing so. 

Though his love for Brooklyn is strong—over half of conversations I’ve witnessed between Downey and his patrons touch on his Brooklyn history—he says he was turned off by the stereotypical attitude that can come with big cities, where people are more apt to act like they don’t remember or know or need anyone they meet in passing, like one’s fellow café-goers.

“So I decided right away that even if, God forbid, I didn’t like you, I wasn’t going to pretend I didn’t know you [or] I didn’t know your order,” Downey says about opening The Victory, first in Brooklyn’s Boerum Hill neighborhood and later in Madison.

Photo of Downey standing behind the counter, a cup of coffee in front of him.
Downey stands behind the counter at The Victory, staring at a latte he just made, on April 28, 2023. Photo by Alyssa Allemand

The first time I came to The Victory in Madison, my friend and I were on an excursion to check out every café in the city and surrounding area we hadn’t yet been to. I’ve loved the Atwood neighborhood since coming to Madison in 2016, and The Victory is one of its staples. Like the neighborhood overall, The Victory is eccentric yet subtle. It doesn’t take itself too seriously. It has fun. It’s “low-key and manic—an unusual combination,” as Downey once told me.

Music is played passionately. By this, I mean, Downey plays whatever he wants.

“We don’t have live music here ever, but we do play music at a volume where you can actually hear it,” he says. “And I have really good speakers and subwoofer, and we make sure that the music quality is really good in terms of what you’re hearing. Whether or not you like the music is subjective, of course, but we do try and cast a fairly wide net.”

“I tell people when I hire them—with hopefully a certain amount of humor, ‘Everything here is about me.’ So, I mean, I run the shop the way I want. I play the music I want to, put stuff on the wall that I want,” Downey says.

The walls are painted shades of yellow gold with big silver diamonds. The ceiling is filled with an intricate metallic gold design. Downey’s own geometric, abstract art hangs throughout the café. Flourishing plants soak up sunlight and fill the counters. The tables are wobbly, there are rips in the chairs, scratches dance on the countertops.

The cold brew is strong and the food menu has your basic sandwiches with some unique twists. But what The Victory offers us most is Downey’s minor cult of personality. 

In my case, The Victory is where I go when I want to feel taken care of. 

One day, I mentioned to Downey that I was bummed I forgot to bring a book with me while I hung out at The Victory to work remotely on a slow day. He went down to the basement and brought up a stack of books for me to peruse through. He’s given me old drawings of his to hang up in my apartment. When I need to take a call or forget I have a Zoom meeting scheduled while working at the café, he lets me take my laptop downstairs and set up shop.

“I guess, at the risk of sounding apart from my usual smartass persona … I like that people come here and feel that it gives them something more than just me being a human coffee vending machine,” Downey says.

Photo of the interior of Victory Cafe. The walls are yellow with silver and gold diamond shapes framing Downey's artwork. A man sits at a small table working on a laptop.
A customer works on a laptop at one of the tables at The Victory. The walls are filled with gold and silver diamonds and feature artwork of Downey’s that can be purchased. Photo by Alyssa Allemand.

He’s a mix of spoken and unspoken gestures. 

Andrea Gonzales-Paul, guitarist and lead vocalist for Madison band Mickey Sunshine, quit her marketing job on a whim after numerous rant sessions with Downey, she says. Gonzales-Paul asked if she’d be able to work at The Victory, without having any prior barista training, and Downey said yes.

“He taught me everything and we had such a lovely summer together,” Gonzales-Paul says. “He basically, like, showed me all this old punk rock music that I wasn’t aware of, and the community was really cool who’d come in. And Patrick is just like an artist in his own right, and it was very inspiring to be around.” 

Downey is humble when it comes to his own musical and artistic work.

“I have played music for most of my life, just mostly for myself, but sometimes in bands,” he says. “And so I’ve never really given up the idea that I might have some song or two to contribute to the culture. So I play music.”

But he keeps a poetic perspective in managing The Victory.

“If you could say that your everyday life is supposed to be something to do with art, then I guess that’s kind of how I approach this,” he says.

In March of this year, Downey purchased the building that houses The Victory after it went up for sale. Downey figured, “Someone’s going to double my rent, so it might as well be me.” 

“We have something to show for scraping along here for 13 years and not making any money,” he says.

The Victory radiates community in a way that is not forced. 

What I pieced together through talking with Downey is that at the root of his seemingly rugged and self-described “smartass persona” is simply a desire to create a space where he can unapologetically be himself. The authenticity Downey harnesses in that act is then shared with those of us who frequent The Victory. 

It’s a space to feel the consistency of being cared for and the euphoria of knowing anything can happen when someone like Downey can effortlessly bring a community together. 

We don’t come to The Victory for the coffee, really. We come for him. 

“Like, it’s my own little world. It’s a little tiny world,” Downey says. “It’s not even—it’s like 14 seats, legally. But it’s my little world, and that’s fine.”

Help us publish more weird, questing, brilliant, feisty, “only on Tone Madison” stories

This site uses cookies to provide you with a great user experience. By continuing to use this website, you consent to the use of cookies in accordance with our privacy policy.

Scroll to Top