By Kirk Spitzer
October 20, 2014

It must have seemed like a good idea at the time.

Prime Minister Shinzo Abe appointed five women to his Cabinet last month in a major shakeup designed to show support for female empowerment and help smooth the way for an unpopular political agenda. But all that unraveled Monday with the abrupt resignation of two of those appointees—Economy, Trade and Industry Minister Yuko Obuchi and Justice Minister Midori Matsushima—for campaign spending violations.

The controversies could not have come at a worse time for Abe. His economic policies are faltering and his Cabinet approval ratings had dropped below 50 percent even before the spending scandal broke last week. Abe faces tough decisions within the next few months on policy issues ranging from restarting nuclear reactors to imposing a second round of tax hikes. He’s also struggling to repair relations with China and South Korea over historical issues and territorial disputes, even as the Asia-Pacific Economic Cooperation summit in Beijing next month looms.

“Abe no longer seems the invincible Superman that some had imagined, and that weakens him both domestically and in Japan’s diplomatic dealings,” says Jeff Kingston, director of Asian Studies at Temple University Japan. “On all of his signature policies — ranging from nuclear reactor re-starts to arms exports, collective self defense and state secrecy legislation—a majority of the public is opposed.”

Trade Minister Obuchi and Justice Minister Matsushima submitted their resignations Monday. They were the first Cabinet members to step down since Abe took office in December 2012—a remarkable period of stability in Japanese politics, where ministers not infrequently are called upon to fall on their sword. It was also a reminder of Abe’s scandal-plagued and inefficient first term in 2006-7, which ended after barely a year. A pension records scandal and the suicide of his agriculture minister during an expense-spending probe, along with poor health for the Prime Minister himself, helped doom Abe’s first go-around.

Obuchi, 40, was accused of funneling campaign money to her sister and brother-in-law and to improperly subsidizing entertainment junkets for supporters. Matsushima stepped down for improperly distributing more than $100,000 worth of paper fans to constituents. Obuchi’s resignation in particular could be a major loss for Abe and the ruling Liberal Democratic Party. A telegenic mother of two, Obuchi had been expected to help Abe with the controversial restart of Japan’s nuclear power plants—a wide majority of the public remains opposed to atomic energy—shut down since the Fukushima disaster in 2011.

Obuchi’s portfolio includes authority over the nation’s nuclear power plants and her softer image—a young mother, after all—was expected to soothe public anxiety over plans to restart the reactors. Obuchi is the daughter of former Prime Minister Keizo Obuchi, who ran Japan from July 1998 to April 2000, and had even been touted as a possible successor to Abe somewhere down the road. But the close scrutiny that comes with a Cabinet appointment exposed her as a political lightweight and a product of the LDP machine, says Michael Cucek, a researcher and author of a respected political blog in Tokyo. “She represents someone who vaulted into prominence by the death of a sitting prime minister, taking over the family business without ever knowing much about how the whole machine works,” he said.

And that may not be the end of it. The remaining three female appointees have drawn heavy criticism, or worse, for alleged connections to neo-Nazi or right-wing fringe organizations, or for visiting the war-linked Yasukuni Shrine. A 2011 photo of Internal Affairs Minister Sanae Takaichi posing with the leader of the National Socialist Japanese Workers Party was discovered on the group’s website shortly after Takaichi’s appointment last month. Postings on Yamada’s blog seem to profess admiration for Adolf Hitler, and videos posted on the website show Yamada and group members wearing stylized swastikas. Takaichi said she was unaware of Yamada’s affiliation when the photo was taken and that it had been posted to the group’s website without her knowledge. She said she asked for the photo to be removed as soon as she learned of it, and that the group complied.

Similarly, a 2009 photo of National Public Safety Commission chief Eriko Yamatani posing with the members of the far- as Zaitokukai group, which has mounted virulent street demonstrations and hate speeches against ethnic Koreans and other foreigners living in Japan. Yamatani also said she was unaware that her photo had been taken with members of the group or that it had been posted online. She said it was taken down at her request after she learned of it.

On Saturday, all the three of the remaining female Cabinet appointees made formal visits to Yasukuni, where 14 convicted “Class A” war criminals—leaders of wartime Japan—are enshrined. That drew a rebuke from China, which remains deeply skeptical of Abe’s revisionist views of history. That visit will complicate Abe’s efforts to repair relations with Japan’s neighbors—and maybe its citizens, says Kingston. “I think there is a great wave of schadenfreude sweeping across East Asia as Abe’s gathering woes weaken his political standing. The Japanese public, too, are happy to see the Abe juggernaut sputtering as Abenomics fizzles and his culture war to redefine national identity backfires.”

Read next: Japan Court Orders Google to Remove Search Results

Contact us at editors@time.com.

SPONSORED FINANCIAL CONTENT

Read More From TIME

EDIT POST