How Immigration Fears in the U.K. Are Fueling Support for an E.U. Exit

David Kirby has had enough. At 71, the retired electrician and lifelong Londoner doesn’t like the way England is changing.

“The country has had it with everything,” he says over a pint on a December day at a J.D. Wetherspoon pub in southeast London. “The way they are letting everybody in, it is too much. It is overloaded now. There are too many foreigners in the country.”

Like many who are broadly opposed to immigration here, Kirby thinks Britain should leave the European Union—and soon, he’ll get the chance to vote for it, in a referendum likely to be held late in 2016. Prime Minister David Cameron pledged in 2013 to put the country’s membership to a vote in part to solidify support on his right flank in the build-up to this year’s general election — though he has said he favors staying in the union under certain terms.

The possibility that Britain might decide to leave the E.U.—dubbed “Brexit”—is as dominant in the British press as the Republican primary is in the U.S., and has eclipsed the rest of Cameron’s agenda. Thursday marks a turning point in that debate as the prime minister travels to Brussels for a summit with the European Union heads of state, in hopes that he can make progress in negotiations over E.U. reforms that he thinks will keep his country in the union. Cameron has said that if the E.U. can agree to his proposed reforms before a summit in February, he will campaign to convince the British people to remain in the E.U. If it does not, he has threatened to campaign for Britain to leave — an outcome that could be a mortal blow to a union already battered by economic headwinds and a historic migrant crisis.

It’s also looking likelier than ever. The share of Brits in favor of ‘Brexit’ has leapt dramatically since the summer, when hundreds of thousands of migrants from the Middle East and north Africa poured into Europe, sparking a continent-wide debate on migration. The Paris attacks probably didn’t help. In June, 61% of Brits favored staying in and 27% wanted to leave, according to Ipsos MORI, a polling firm in the U.K. By October that gap had narrowed to 52% versus 39%. A more recent December poll by the firm ICM Unlimited has it at a dead tie.

Read more: U.K. Edges Toward Departure from European Union

The fervor over immigration means there’s now a “fairly high chance” British people will vote to leave the E.U., says Charles Grant, director of the pro-E.U. Centre for European Reform, and a supporter of Britain remaining in the EU. “They see boatloads of Syrians arriving in Greece and they are afraid that they will one day end up in Britain.”

Although some migrants will no doubt be resettled in the U.K., it will be a trickle rather than a flood. Cameron has agreed to take 20,000 Syrian refugees directly from Syria, and illegal immigration isn’t as big a problem in the U.K. as it is on the continent with its open borders. Although there are camps of migrants in Calais attempting to gain access to the U.K., the English Channel is a challenging obstacle for them to overcome.

The fears likely stem from a more historic frustration with immigration on this island. Anti-immigration sentiment has been widespread at least since the 1950s, when post-war immigration from former British colonies began changing the racial makeup of the country. Today, workers who have lived through years of economic stagnation are more likely to be frustrated at the influx of Eastern European migrant workers since a British policy change in 2004 made it easier for them to come to the U.K. for work. One such example: the U.K’s Office for National Statistics reports that the Polish-born population grew from 75,000 in 2003 to 532,000 in 2010.

Though these workers often are found in poorly-paid agricultural and service jobs, some believe they bring wages down and put pressure on public services. More likely, they simply give people the sense that their island nation is becoming less “British.” Muslim extremism and refugees may have little to do with workers coming in from Poland, but the refugee crisis has caused them to bleed together in people’s minds.

So it’s vital for Cameron to look as if he is making progress on immigration with the E.U. over these talks. But of all of his demands for reforms, immigration reform will be the most difficult to get. Cameron has proposed that migrant workers from E.U. countries who come to work in Britain not be allowed to access certain benefits, such as tax breaks and housing benefits, until they have worked in the country for four years.

Read more: These Are the 5 Facts That Explain the Surprising U.K. Elections

That’s a hard sell for other European Union states, as the free movement of people is a founding principle of the union. The key opposition is likely to come from Britain’s largest neighbors: France and Germany. The French may be disinclined to give Britain concessions on immigration for fear it would stoke populist factions within France, says Raoul Ruparel, the co-director at Open Europe, an independent U.K. think tank that favors a less centralized E.U. And experts say Angela Merkel, the union’s pre-eminent champion, may hold a hard line despite the consequences for Britain’s place in Europe.

“In Merkel’s view, it would be a disaster if Britain left,” says Grant. “On the other hand, she and her officials all believe that the single market is a package deal, in terms of free movement of people, goods, and services. I don’t believe she is, at the moment, willing to contemplate budging on that fundamental principle.”

Political analysts agree immigration could be a deal-breaker in the referendum. “Cameron has to get something on immigration,” says Ruparel. “It would be hard for him to campaign to stay in and bring his party with him, if he can’t show that the E.U. can make some progress on that issue.”

The stakes are high. There’s little question that ‘Brexit’ would be a disaster for the European Union, weakening its influence on the world stage and putting its future in doubt. But it’s also an open question whether Brits would vote to quit the E.U., when push comes to shove. Pragmatism has been known to overcome xenophobia and isolationism at the ballot box — as French voters proved by ultimately shunning the far-right, anti-immigration Front National in the second round of regional elections last weekend.

Cameron now has until February to find a compromise on reform that is enough for him to take back to his people. “The question that everyone has been asking is whether there is a sweet spot to satisfy enough of the British population and be realistic to EU partners,” says Hans Kundnani, a fellow at the German Marshall Fund in Berlin. “No one knows the answer to that yet, but I think he might just pull it off.”