Orange Is the New Black: The Time Martha Stewart Almost Went to Litchfield

Many fans of Orange Is the New Black, which returns Friday with its 13-episode second season, know there are a number of differences between Netflix’s breakout hit and Piper Kerman’s original memoir of the same name. One of the more interesting stories left on the cutting room floor of Jenji Kohan’s television adaptation was the time Kerman nearly crossed paths with Martha Stewart, who seemed to be headed to Danbury — the real-life prison Orange’s Litchfield is based on —following her insider-trader scandal in the early 2000s.

It never actually happened. But what if the show did add a Martha Stewart-like character to the show? The arrival of the queen of all things home living would embody so much of what makes Orange Is the New Black addicting: inmate clashes, critiques of America’s prison system and a look at prisoners’ tumultuous relationship with the outside world.

When Martha Stewart was sentenced in 2004 to five months in prison and five months of home confinement, she asked judges to send her to Danbury’s federal correctional institution so that Stewart’s elderly mother could visit her from her home in Connecticut. News of Stewart’s possible arrival was a major event for the women of Danbury, who were “deflated” when she was ultimately sent to a prison in West Virginia. “Everyone had been hoping that against all odds she’d be sent to live with us, either because they believed that her presence would somehow raise all our boats or just for the entertainment value,” Kerman writes in her book.

They weren’t the only ones excited. Helicopters frequently flew over the camp during that time, and reporters tried to sneak their way onto the facilities. Kerman says it was no accident that Stewart ultimately didn’t serve her time at the Connecticut prison: She suggests the authorities weren’t thrilled about having such a high-profile inmate with a major media presence inside the prison, and so Danbury went out of its way to keep Stewart’s Connecticut wish from coming true. “The Camp had been ‘closed’ to new inmates since her conviction, allegedly ‘full,’ although we had more empty beds as every week passed,” Kerman writes.

It’s easy to imagine the type of trouble Stewart — who told Vanity Fair that her prison nickname was “M. Diddy” — would get into if a fictionalized version of her made it to Litchfield. She might butt heads with prison officers about regulations for decorating her bunk — or maybe she’d have strong feelings about the aesthetics of the holiday pageant. “I imagined Martha Stewart trying to take over Pop’s kitchen,” Kerman writes in the book. “That would be better than Godzilla vs. Mothra.” (Not that Pop’s Netflix equivalent, Galina “Red” Reznikov, played by Kate Mulgrew, needs any more enemies — the show’s second season finds her clashing with old rival and notorious drug-dealer, Yvonne “Vee” Parker, played by Lorraine Toussaint.)

Although the Orange’s best drama happens entirely inside the prison confines, a few of the first season’s most poignant moments occurred when the outside world got involved — like when Larry (Jason Biggs) goes on a This American Life-esque show to talk about his fiancée’s incarceration and horrifies Piper’s bunk mates by revealing what she really thought about them. Orange was partly about how women find meaning and pride in a system that deprives them of so much freedom and dignity, and a Martha Stewart story would continue to explore the ways inmates’ self-esteems are tested. Kerman writes that some of her fellow prisoners were deeply hurt when an old PEOPLE magazine story about what Stewart could expect from prison quoted a source who called Stewart’s potential new roommates “the scum of the earth.”

As for the fun part—who would play Stewart? — the current cast might spark some ideas. In an interview with TIME hitting newsstands this week, Taylor Schilling, who plays Piper, says noted lover of crafts Diane Keaton is the latest addition to her list of dream co-stars, and the resemblance is already there. But there’s also a strong case against creating a Stewart-like character — because Orange Is the New Black already has its fish-out-of-water white lady, Piper.

That’s not to say a Stewart character’s struggles would have been identical to Kerman’s or any less interesting, but as Orange creator Jenji Kohan has said, the character of Piper Chapman was “a trojan horse,” a way to tell the stories of women of all identities who normally don’t get TV shows made about them. As the second season shifts the focus away from Piper and shares the love with the other inmates, a hypothetical Martha Stewart character, while amusing, might just distract from the real draw of the show: the women who were already there.