Gilmore Girls: Jared Padalecki, Milo Ventimiglia talk Dean and Jess

During Gilmore Girls‘ original seven-season run, there weren’t many instances of violence. Typically, the biggest fight in any episode would involve Emily Gilmore screaming at her daughter, or, even more likely, her maid. But there was one big fight scene in the show’s third season.

Soon after Rory officially started dating Jess, she found herself at a house party. But what she didn’t know at the time was that Jess’ sour mood stemmed from the fact that he’d just found out he wasn’t going to graduate high school (which also meant he couldn’t take his girlfriend to the prom). So when Jess took his anger out on Rory and she ran away crying, it was Dean who stopped her. And after months of Dean and Jess fighting over Rory, the two of them finally duked it out. Literally.

“We filmed that scene at the Warner Bros. ranch, where they filmed the intro to Friends and a lot of shows,” Jared Padalecki tells EW. “But we were so used to shooting Stars Hollow at the Warner Bros. lot, it felt like a really unique stand-alone scene and I remember it vividly. I remember filming it, blocking it, my stunt guy, the scene on the lawn. I was really, really excited about it.”

Watch EW Reunites: Gilmore Girls here now on the new People/Entertainment Weekly Network (PEN). Go to PEOPLE.com/PEN, or download the PEN app on Apple TV, Roku Players, Amazon Fire TV, Xumo, Chromecast, iOS, and Android devices.

What many fans will remember about the scene is the fact that “bad boy Jess” had a bit of a disadvantage when fighting “sweet Dean,” and it wasn’t because Padalecki couldn’t physically feel anything. “I can’t remember if I was shooting the MacGyver pilot — whatever happened, I was injured,” Padalecki recalls. “I remember being on like four Advils every four hours and just not really feeling anything. And I remember Milo [Ventimiglia] going like, ‘This is kind of ridiculous because you’re twice my size and I’m holding my own.'”

It’s a sentiment Ventimiglia still shares, even today. “The funny thing that I always think about is, in Hollywood, usually people are pretty matched up. But Jared is 6-foot-4, I’m 5-foot-10. There’s a difference in reach and weight between us. So I usually like to say what Dean had in size and reach, Jess matched in crazy.”

And yet, the thing Ventimiglia remembers most is having to fight a friend. “Those are always funny moments to play when you have to get into a fist fight with someone that you absolutely adore,” he says. “It was a lot of Jared and I just making sure that A) we were safe, but B) it looked aggressive.”

Thankfully, they achieved that.