January 6, 2015 6:13 PM EST

At 11:52 a.m. Tuesday, Virginia Republican Rep. Scott Rigell was watching C-SPAN in the Capitol. He was still thinking of supporting Speaker John Boehner’s re-election, when he saw a backbencher’s name splash across the screen as a potential challenger. Rigell then sought out the upstart—Florida Republican Rep. Daniel Webster—on the House floor and confirmed that he would serve if he won. Even though Rigell knew the likelihood of that happening was low, he ended up voting for Webster a few hours later.

“To go to from voting for the Speaker to voting for Mr. Webster there’s an intense amount of thought in a relatively short amount of time,” he said.

In the biggest defection in at least 100 years, according to the Washington Post, 25 Republicans voted against Boehner, who nonetheless went on to win a third term as Speaker. Most of the opposition came from the right of Rigell, with representatives like Louie Gohmert and Ted Yoho, two other members who officially ran against Boehner and might as well carry pitchforks when they stride into the Capitol.

But while the anti-Boehner crowd is loud and can sometimes move bills to the right, it lacks a leader. Yoho told TIME he jumped in only after an unnamed conservative who could corral their corps wouldn’t. And Webster, who got 12 votes, didn’t whip anyone to vote for him, according to Rigell, and Gohmert and Yoho received only five votes total.

For his part, Rigell’s vote was “simply [a] connection between the calendar and the appropriations bills.” He wants a Speaker which would cancel extended breaks like the summer recess “unless and until” all 12 appropriations bills have been voted upon. He holds Boehner in “high regard” for lowering discretionary spending, but talks of changing the institution itself. Of course, Boehner isn’t the only to blame; while the appropriations process has been largely negotiated behind close doors by the top congressional leaders, it has rarely passed all 12 of the aforementioned bills in decades.

Still, Boehner’s allies noted that the vast majority of House Republicans—216—voted for Boehner.

“Under his leadership in this new Congress we will pass strong, conservative legislation which will make its way to the president’s desk,” said Louisiana Rep. Steve Scalise, the GOP Whip, in a statement. “Speaker Boehner’s commitment to bold, conservative solutions that reflect the priorities of the American people will move America forward and help get our country back on the right track.”

More Must-Read Stories From TIME

Contact us at letters@time.com.

Read More From TIME
Related Stories