Tara Lipinski Opens Up About 'Heartbreaking' Miscarriages During Infertility Struggles: 'So Isolating' (Exclusive)

Tara Lipinski is opening up about her own journey to growing her family.

Speaking exclusively with PEOPLE, the Olympic figure skater, 41, reveals that over the past five years, she and husband Todd Kapostasy have been "on the fertility journey from hell."

"I have never really publicly spoken about it," Lipinski tells PEOPLE of her experience with infertility. "So it just felt like at this point I finally felt ready."

"I grew up as an Olympic athlete and at a very young age I was in the spotlight and sharing my life publicly," she explains. "This was just the first time in my life — and it's gone on for so long — that I felt like I was living this lie or living with this big secret."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

"It's been five years of my life and my husband's life and not speaking about it. I just wasn't emotionally or mentally able to. I was so overwhelmed and struggling day-to-day that I wasn't in a place to bring other people in and that thought really scared me."

Despite being overwhelmed, Lipinski knew that it was important to share her story. "All along I knew that I really, really wanted to talk about my story because infertility in itself is so isolating. I think we hear so many more stories now about IVF and we know people use surrogates and we hear this overarching view of it. But you never hear those deep details. And I think when I started this journey, that's all I wanted — to connect with women."

Lipinski has a close-knit family and group of friends that she's been able to lean on throughout her journey with infertility. Still, the athlete notes that even with the support the time has felt like a solitary experience.

"I feel like when you're going through infertility, it is a very lonely place, and not everybody has gone through it, not everyone understands how to support you or how to be there for you," she tells PEOPLE.

"I think for us, infertility, this journey to build a family became a full-time job. It consumed my life. Every waking moment was somehow circulating around fertility and the next appointment and the next doctor, the next cycle and grief and loss and it was a never-ending journey."

While Lipinski's always wanted to be a mom, she explains she never thought she'd struggle to get pregnant.

"I was so focused on my career and my work and I met my husband a little bit later in life, and I just never really thought about the next steps," she says. "I knew I always wanted to be a mom. I just didn't think there was a time limit on it."

Once she started thinking about becoming a parent, Lipinski realized it wasn't going to be as simple as she might have thought.

"We didn't succeed in this world of IVF very easily," she tells PEOPLE. "The idea of IVF is wonderful because it gives women who have medical issues and are struggling with the inability to conceive an opportunity to bypass that and to potentially build a family."

"When I started IVF, I thought, 'How amazing is this, we're just going to have a little football team of embryos there waiting for us whenever we decide to get pregnant,' " Lipinski recalls thinking. "And that certainly was not the case when we started. It's been a long and winding journey full of failure after failure after failure."

"It's just been years of facing almost every obstacle you can in the world of IVF," she says of her experience. "There's so many different moments that I look back on where I thought, 'Oh, we're there, we're nearing the end.' And it just never got there for us."

"In the last five years, I've been under anesthesia 24 times, had four miscarriages, four D&Cs [dilation and curettage], six failed transfers, eight retrievals and was diagnosed with endometriosis with two subsequent major surgeries," she tells PEOPLE. "My life revolved around doctor appointments and procedures."

"My lowest point was last summer, having two back-to-back consecutive miscarriages. One miscarriage was heartbreaking but four made me feel like a shell of myself. After the second miscarriage, I stopped crying. There were no more tears left. I just felt numb."

It wasn't just the miscarriages that affected her; Lipinski explains that the failed IVF transfers were also a remarkable loss. "We hit so many barriers with 'loss.' There is a grief to not only the miscarriages but also to the failed IVF transfer attempts."

"Retrievals and getting embryos on ice is not an easy process," the athlete says. "A lot of hormones and emotions are part of that process. You are not guaranteed to walk away with any embryos but when you do get them, they are precious. You envision them becoming your potential child, so to lose that chance in a transfer is heartbreaking as well."

Though she was an Olympian for most of her early life, the professional figure skater explains that the intensity of the sport was nothing compared to her journey with IVF. "People will always talk with me about being an Olympian and how hard the Olympics were, what was that like? How did you do the impossible? And in my mind, it doesn't seem like the impossible at all."

"If I thought the Olympics was a difficult journey, I had no idea what was in store for me because this is by far the most difficult thing that I've ever encountered in my life," she reveals.

"To get through this has really shown me how strong I am. You think as an athlete, 'Oh, I'm so strong and am able to get through anything,' and little did I know that this journey would actually teach me that I could do that. I was really put up to the test so many times where I thought, 'Oh, I have fallen down so hard I don't know how I'm going to get back up.' "

Despite the tough times, Lipinski never gave up hope. "Through it all, I haven't given up," she says. "My urge to become a mother is what had gotten me through and has kept me going. I've never wanted anything more."

The athlete and her husband will open up about their infertility struggles in a new podcast. Lipinski, who explains that she'd go on "deep Google searches and Instagram black holes" to find women with similar stories, says that in the podcast, "we go on the journey with our listeners and we take you through every single event that happened to us."

The Olympian's husband, who is a documentary film director, joins her on the podcast as they share both of their perspectives with listeners. "We felt like let's do this together. We will produce it, direct it, buy everything we need to set up a studio in our home where we're comfortable and we feel that we're in this safe place to talk and bring up all these memories together."

"We didn't want to bring in anyone else. We didn't want someone to take care of audio or the camera. We wanted to feel like this was our thing and that we could get to a super vulnerable place. Obviously this has been pretty traumatic for me, so I definitely needed to do it at home and to be able to just do it with Todd."

Lipinski added: "It just felt really nice that we could do it on our own terms, tell our story the way we wanted to, and to document this time in our life, and I think a little bit to maybe heal while we do this."

Tara Lipinski: Unexpecting premieres on August 29.