Fact Vs. Fiction: Princess Margaret's 'Raunchy' Marriage (Affairs! A Love Child!) on 'The Crown'

There is much more to know about Princess Margaret than her doomed love affair, her taste for liquor and her epic put-downs.

Now a new generation of royals fans will be introduced to her wild marriage to society photographer Antony Armstrong-Jones, a central theme of season two of The Crown.

Margaret (Vanessa Kirby) married Armstrong-Jones (Downton Abbey‘s Matthew Goode) in May 1960 — but even while they planned their engagement, the groom-to-be was still conducting another affair.

As he prepared to head to Queen Elizabeth's Scottish retreat of Balmoral in the fall of 1959, Armstrong-Jones was sleeping with Camilla Fry, who was then married to one of his oldest friends, Jeremy Fry.

Camilla’s daughter Polly was born in May 1960, just three weeks into Margaret and Armstrong-Jones’s honeymoon. But despite society rumors, it was only at the beginning of this century that Polly learned her true parentage through a DNA test. (Yes, Armstrong-Jones was her biological father.)

Elizabeth and Philip: A Royal Romance—The Real-Life Story Behind ‘The Crown’ is available now.

The society photographer’s complex love life didn’t end there. Addressing rumors of his bisexuality, he once said, “I didn’t fall in love with boys, but a few men have been in love with me.”

Armstrong-Jones, who was titled Lord Snowdon by the Queen a year after his wedding, was a bohemian soul – and is the subject of the “raunchiest scenes in the show,” says royal historian Robert Lacey, author of the official companion book for the first season. “You see how Margaret was a victim of Tony Armstrong-Jones even before they got married,” he tells PEOPLE. “It captures the feeling of the times.”

For much more on The Crown, pick up this week’s issue of PEOPLE, on newsstands Friday

“It is very explicit about the promiscuity of Armstrong-Jones — and Margaret wasn’t aware of it. That will open a lot of people’s eyes.”

The son of a senior lawyer and a society hostess, Armstrong-Jones was the first commoner to marry a member of the royal family in 400 years. When he and Margaret divorced after 18 years, the split would signal something new for the family too: Their royal divorce preceded those of the Queen’s children in the latter part of the century.

Watch Elizabeth and Philip: A Royal Romance now on the new PeopleTV network. Go to PeopleTV.com, or download the PeopleTV app on your favorite mobile or connected TV device.

Margaret is said to have met the photographer in 1958, and she became a subject for his photographs and enjoyed being ferried off to his studio in east London away from the stiff protocol of royal life.

Kirby tells PEOPLE in this week’s issue that he was the “antithesis” of “all the formality and stuffiness of her family.”

“It was the world that we got to explore in this new London set, and going to the more dangerous side that we haven’t seen before,” she notes.

Kirby believes that her character never got over doomed love affair with Group Capt. Peter Townsend, whose previous divorce was frowned upon by the Queen and the Church of England. “It always rang true that if you waited for somebody for that long, especially when you could have had many different people and marriage was that important to you, I can’t imagine that it wouldn’t hurt forever really,” she says.

Margaret is said by some biographers to have married Armstrong-Jones as an immediate reaction to the heartbreak of her split from Townsend — something her biographer Christopher Warwick, who wrote Princess Margaret: A Life of Contrasts, disputes. He says he once asked her if it was true that it was a rebound romance and she replied, “I married Tony five years after the end of Peter Townsend.”

What is clear is that she had been informed of Townsend’s wishes to marry someone else — a woman who happened to be 10 years Margaret’s junior — in October 1959 via letter while she was staying at Balmoral – with Armstrong-Jones. The former lovers had previously made a pact never to marry another , and Craig Brown, in the current biography Ma’am Darling, reports that Margaret’s letter back “was immediate, passionate and furious.”

The decision to marry Margaret was a difficult one for Armstrong-Jones, says Brown. “There was so much he would have to give up, not least his other girlfriends, one of whom confided to a friend that Tony had wept on her bare breasts at the prospect of getting married to royalty.”

As Margaret and Snowdon eventually settled into married life, the swinging ’60s dawned and the couple were at the heart of an arty world that included actor Peter Sellers and The Beatles. “Stories about them were legion, with their star-studded parties at Kensington Palace,” says Warwick. “If you were invited by them, you were being invited to breathe in rarified air.”

Margaret and Armstrong-Jones divorced in 1978 but remained on good terms until her death in 2002. In later life, she was famous for her affair with playboy Roddy Llewellyn and her bon vivant vacations on Mustique.

Snowdon died peacefully at home in January this year.