The Woman Who Spied on Confederates In Jefferson Davis’ Own Home

This piece is part of an ongoing series on the unsung women of history. Read more here.

After the Civil War, an unusual public figure took to the stage in New York. Her name was Richmonia Richards, and she delivered a scintillating account of wartime feats of espionage that must have delighted her Northern viewers. But two things about Richards stood out—the name was an alias, and she was a woman of color. Both borrowed names and black skin play into the amazing story of Mary Richards Bowser, a woman now recognized as a Union spy who actually made her way into the Confederate White House during the Civil War.

Bowser was born a slave in Richmond around 1839. She was raised in the home of Virginia’s prominent Van Lews, a family with Northern roots who nonetheless owned slaves. Elizabeth Van Lew, Bowser’s owner, was Quaker-educated and loyal to the abolitionist cause. When her father died in 1843, the Van Lews freed all of their slaves, and Elizabeth later used money from her father’s estate to buy and free relatives and friends of the people she had freed.

One of those freed slaves was Bowser. Elizabeth took note of the young woman’s intelligence and paid to send her to a Quaker school for black students in Philadelphia—partly because it was illegal for a freed slave to stay in Virginia. Under an 1806 laws, they had to leave the state within a year or run the risk of being re-enslaved.

Get your history fix in one place: sign up for the weekly TIME History newsletter

Perhaps in response to that persistent threat, Bowser soon headed to Africa, to live in Liberia. At the time, there was increasing pressure for former slaves to leave the U.S. and settle there, and perhaps Bowser sought a new life in a place where she did not face the threat of the pervasive racism that faced even free blacks in the abolitionist North. But for whatever reason, Bowser did not stay in Africa. Instead, she returned to Richmond illegally.

Back at home with the Van Lews, Bowser remained with the family as a free servant. But Virginia would not recognize her freedom and arrested her for “claiming to be a free person of color.” Elizabeth paid her bail and claimed that Bowser was in fact a slave, laying the foundation for one of the Civil War’s greatest ruses.

Elizabeth wasn’t just an abolitionist—by the time the Civil War began, she was a committed supporter of the Union cause. Unlike many of her counterparts, she felt that her best work on behalf of the North could be done in the heart of the Confederate capital. She remained in Richmond, spending her time doing good deeds among the imprisoned Union soldiers at the “rat hell” of Libby Prison. Elizabeth herself was able to escape the notice of her Southern enemies by posing as a mentally ill figure who became known as “Crazy Bet.” It was easy for Richmonders to overlook her eccentricities…and miss the fact that she was running a spy ring in the center of the Confederate capital.

MORE: The Special Reason Harriet Tubman Is Perfect for the $20 Bill

But she wasn’t alone. While Elizabeth Van Lew had to feign insanity to remain in Richmond without arousing suspicion about her abolitionist views, Mary Richards Bowser’s stupidity was assumed by Southerners who did not allow black people educations or dignity. Aided by those prejudices and a photographic memory, Bowser posed as a dim-witted slave and began spying on Confederate officials. Van Lew encouraged a friend to take her to events and eventually Bowser—in the role of a slave who had been hired out by her master—was employed full-time by Jefferson Davis himself. Her time working in the house of the Confederate president— reporting all the while on the comings and goings of the household—was by far her biggest coup.

“When I open my eyes in the morning, I say to the servant, ‘What news, Mary?’ and my caterer never fails!” Van Lew wrote in her diary during the war. “Most generally our reliable news is gathered from negroes, and they certainly show wisdom, discretion and prudence which is wonderful.”

After the war, Bowser took on the persona of “Richmondia Richards” and other aliases, speaking to New Yorkers about her spy work and spreading the word about her educational work with freed slaves. Her speeches were sarcastic, humorous and filled with eloquence about the equal treatment she felt was finally black peoples’ due after the long years of enslavement and toil. Soon Bowser gave history the slip, too. It’s not certain what happened to her after the 1860s—perhaps she changed names or simply faded into the woodwork. She left behind only tales of her bravery and questions about what became of a Union spy who played on prejudice to help the anti-slavery cause.