The Weight Loss Trap: Why Your Diet Isn’t Working

Like most people, Kevin Hall used to think the reason people get fat is simple.

“Why don’t they just eat less and exercise more?” he remembers thinking. Trained as a physicist, the calories-in-vs.-calories-burned equation for weight loss always made sense to him. But then his own research–and the contestants on a smash reality-TV show–proved him wrong.

Hall, a scientist at the National Institutes of Health (NIH), started watching The Biggest Loser a few years ago on the recommendation of a friend. “I saw these folks stepping on scales, and they lost 20 lb. in a week,” he says. On the one hand, it tracked with widespread beliefs about weight loss: the workouts were punishing and the diets restrictive, so it stood to reason the men and women on the show would slim down. Still, 20 lb. in a week was a lot. To understand how they were doing it, he decided to study 14 of the contestants for a scientific paper.

Hall quickly learned that in reality-TV-land, a week doesn’t always translate into a precise seven days, but no matter: the weight being lost was real, speedy and huge. Over the course of the season, the contestants lost an average of 127 lb. each and about 64% of their body fat. If his study could uncover what was happening in their bodies on a physiological level, he thought, maybe he’d be able to help the staggering 71% of American adults who are overweight.

What he didn’t expect to learn was that even when the conditions for weight loss are TV-perfect–with a tough but motivating trainer, telegenic doctors, strict meal plans and killer workouts–the body will, in the long run, fight like hell to get that fat back. Over time, 13 of the 14 contestants Hall studied gained, on average, 66% of the weight they’d lost on the show, and four were heavier than they were before the competition.

That may be depressing enough to make even the most motivated dieter give up. “There’s this notion of why bother trying,” says Hall. But finding answers to the weight-loss puzzle has never been more critical. The vast majority of American adults are overweight; nearly 40% are clinically obese. And doctors now know that excess body fat dramatically increases the risk of serious health problems, including Type 2 diabetes, heart disease, depression, respiratory problems, major cancers and even fertility problems. A 2017 study found that obesity now drives more early preventable deaths in the U.S. than smoking. This has fueled a weight-loss industry worth $66.3 billion, selling everything from diet pills to meal plans to fancy gym memberships.

For a limited time, TIME is giving all readers special access to subscriber-only stories. For complete access, we encourage you to become a subscriber. Click here.

It’s also fueled a rise in research. Last year the NIH provided an estimated $931 million in funding for obesity research, including Hall’s, and that research is giving scientists a new understanding of why dieting is so hard, why keeping the weight off over time is even harder and why the prevailing wisdom about weight loss seems to work only sometimes–for some people.

What scientists are uncovering should bring fresh hope to the 155 million Americans who are overweight, according to the U.S. Centers for Disease Control and Prevention. Leading researchers finally agree, for instance, that exercise, while critical to good health, is not an especially reliable way to keep off body fat over the long term. And the overly simplistic arithmetic of calories in vs. calories out has given way to the more nuanced understanding that it’s the composition of a person’s diet–rather than how much of it they can burn off working out–that sustains weight loss.

They also know that the best diet for you is very likely not the best diet for your next-door neighbor. Individual responses to different diets–from low fat and vegan to low carb and paleo–vary enormously. “Some people on a diet program lose 60 lb. and keep it off for two years, and other people follow the same program religiously, and they gain 5 lb.,” says Frank Sacks, a leading weight-loss researcher and professor of cardiovascular disease prevention at the Harvard T.H. Chan School of Public Health. “If we can figure out why, the potential to help people will be huge.”

Hall, Sacks and other scientists are showing that the key to weight loss appears to be highly personalized rather than trendy diets. And while weight loss will never be easy for anyone, the evidence is mounting that it’s possible for anyone to reach a healthy weight–people just need to find their best way there.

Dieting has been an American preoccupation since long before the obesity epidemic took off in the 1980s. In the 1830s, Presbyterian minister Sylvester Graham touted a vegetarian diet that excluded spices, condiments and alcohol. At the turn of the 20th century, it was fashionable to chew food until liquefied, sometimes up to 722 times before swallowing, thanks to the advice of a popular nutrition expert named Horace Fletcher. Lore has it that at about the same time, President William Howard Taft adopted a fairly contemporary plan–low fat, low calorie, with a daily food log–after he got stuck in a White House bathtub.

The concept of the calorie as a unit of energy had been studied and shared in scientific circles throughout Europe for some time, but it wasn’t until World War I that calorie counting became de rigueur in the U.S. Amid global food shortages, the American government needed a way to encourage people to cut back on their food intake, so it issued its first ever “scientific diet” for Americans, which had calorie counting at its core.

In the following decades, when being rail-thin became ever more desirable, nearly all dieting advice stressed meals that were low calorie. There was the grapefruit diet of the 1930s (in which people ate half a grapefruit with every meal out of a belief that the fruit contained fat-burning enzymes) and the cabbage-soup diet of the 1950s (a flatulence-inducing plan in which people ate cabbage soup every day for a week alongside low-calorie meals).

The 1960s saw the beginning of the massive commercialization of dieting in the U.S. That’s when a New York housewife named Jean Nidetch began hosting friends at her home to talk about their issues with weight and dieting. Nidetch was a self-proclaimed cookie lover who had struggled for years to slim down. Her weekly meetings helped her so much–she lost 72 lb. in about a year–that she ultimately turned those living-room gatherings into a company called Weight Watchers. When it went public in 1968, she and her co-founders became millionaires overnight. Nearly half a century later, Weight Watchers remains one of the most commercially successful diet companies in the world, with 3.6 million active users and $1.2 billion in revenue in 2016.

What most of these diets had in common was an idea that is still popular today: eat fewer calories and you will lose weight. Even the low-fat craze that kicked off in the late 1970s–which was based on the intuitively appealing but incorrect notion that eating fat will make you fat–depended on the calorie-counting model of weight loss. (Since fatty foods are more calorie-dense than, say, plants, logic suggests that if you eat less of them, you will consume fewer calories overall, and then you’ll lose weight.)

That’s not what happened when people went low fat, though. The diet trend coincided with weight gain. In 1990, adults with obesity made up less than 15% of the U.S. population. By 2010, most states were reporting obesity in 25% or more of their populations. Today that has swelled to 40% of the adult population. For kids and teens, it’s 17%.

Research like Hall’s is beginning to explain why. As demoralizing as his initial findings were, they weren’t altogether surprising: more than 80% of people with obesity who lose weight gain it back. That’s because when you lose weight, your resting metabolism (how much energy your body uses when at rest) slows down–possibly an evolutionary holdover from the days when food scarcity was common.

What Hall discovered, however–and what frankly startled him–was that even when the Biggest Loser contestants gained back some of their weight, their resting metabolism didn’t speed up along with it. Instead, in a cruel twist, it remained low, burning about 700 fewer calories per day than it did before they started losing weight in the first place. “When people see the slowing metabolism numbers,” says Hall, “their eyes bulge like, How is that even possible?”

The contestants lose a massive amount of weight in a relatively short period of time–admittedly not how most doctors recommend you lose weight–but research shows that the same slowing metabolism Hall observed tends to happen to regular Joes too. Most people who lose weight gain back the pounds they lost at a rate of 2 to 4 lb. per year.

For the 2.2 billion people around the world who are overweight, Hall’s findings can seem like a formula for failure–and, at the same time, scientific vindication. They show that it’s indeed biology, not simply a lack of willpower, that makes it so hard to lose weight. The findings also make it seem as if the body itself will sabotage any effort to keep weight off in the long term.

But a slower metabolism is not the full story. Despite the biological odds, there are many people who succeed in losing weight and keeping it off. Hall has seen it happen more times than he can count. The catch is that some people appear to succeed with almost every diet approach–it just varies from person to person.

“You take a bunch of people and randomly assign them to follow a low-carb diet or a low-fat diet,” Hall says. “You follow them for a couple of years, and what you tend to see is that average weight loss is almost no different between the two groups as a whole. But within each group, there are people who are very successful, people who don’t lose any weight and people who gain weight.”

Understanding what it is about a given diet that works for a given person remains the holy grail of weight-loss science. But experts are getting closer.

For the past 23 years, Rena Wing, a professor of psychiatry and human behavior at Brown University, has run the National Weight Control Registry (NWCR) as a way to track people who successfully lose weight and keep it off. “When we started it, the perspective was that almost no one succeeded at losing weight and keeping it off,” says James O. Hill, Wing’s collaborator and an obesity researcher at the University of Colorado. “We didn’t believe that was the case, but we didn’t know for sure because we didn’t have the data.”

To qualify for initial inclusion in the registry, a person must have lost at least 30 lb. and maintained that weight loss for a year or longer. Today the registry includes more than 10,000 people from across the 50 states with an average weight loss of 66 lb. per person. On average, people on the current list have kept off their weight for more than five years.

The most revealing detail about the registry: everyone on the list has lost significant amounts of weight–but in different ways. About 45% of them say they lost weight following various diets on their own, for instance, and 55% say they used a structured weight-loss program. And most of them had to try more than one diet before the weight loss stuck.

The researchers have identified some similarities among them: 98% of the people in the study say they modified their diet in some way, with most cutting back on how much they ate in a given day. Another through line: 94% increased their physical activity, and the most popular form of exercise was walking.

“There’s nothing magical about what they do,” says Wing. “Some people emphasize exercise more than others, some follow low-carb diets, and some follow low-fat diets. The one commonality is that they had to make changes in their everyday behaviors.”

When asked how they’ve been able to keep the weight off, the vast majority of people in the study say they eat breakfast every day, weigh themselves at least once a week, watch fewer than 10 hours of television per week and exercise about an hour a day, on average.

The researchers have also looked at their attitudes and behavior. They found that most of them do not consider themselves Type A, dispelling the idea that only obsessive superplanners can stick to a diet. They learned that many successful dieters were self-described morning people. (Other research supports the anecdotal: for some reason, night owls tend to weigh more than larks.) The researchers also noticed that people with long-term weight loss tended to be motivated by something other than a slimmer waist–like a health scare or the desire to live a longer life, to be able to spend more time with loved ones.

The researchers at the NWCR say it’s unlikely that the people they study are somehow genetically endowed or blessed with a personality that makes weight loss easy for them. After all, most people in the study say they had failed several times before when they had tried to lose weight. Instead they were highly motivated, and they kept trying different things until they found something that worked for them.

“Losing weight and keeping it off is hard, and if anyone tells you it’s easy, run the other way,” says Hill. “But it is absolutely possible, and when people do it, their lives are changed for the better.” (Hill came under fire in 2015 for his role as president of an obesity think tank funded by Coca-Cola. During his tenure there, the NWCR published one paper with partial funding from Coca-Cola, but the researchers say their study, which Hill was involved in, was not influenced by the soda giant’s financial support.)

Hill, Wing and their colleagues agree that perhaps the most encouraging lesson to be gleaned from their registry is the simplest: in a group of 10,000 real-life biggest losers, no two people lost the weight in quite the same way.

The Bariatric Medical Institute in Ottawa is founded on that thinking. When people enroll in its weight-loss program, they all start on the same six-month diet and exercise plan–but they are encouraged to diverge from the program, with the help of a physician, whenever they want, in order to figure out what works best for them. The program takes a whole-person approach to weight loss, which means that behavior, psychology and budget–not just biology–inform each person’s plan.

“We have a plan that involves getting enough calories and protein and so forth, but we are not married to it,” says Dr. Yoni Freedhoff, an obesity expert and the medical director of the clinic. “We try to understand where people are struggling, and then we adjust. Everyone here is doing things slightly differently.”

In most cases, people try a few different plans before they get it right. Jody Jeans, 52, an IT project manager in Ottawa, had been overweight since she was a child. When she came to the clinic in 2007, she was 5 ft. 4 in. tall and weighed 240 lb. Though she had lost weight in her 20s doing Weight Watchers, she gained it back after she lost a job and the stress led her to overeat. Jeans would wake up on a Monday and decide she was starting a diet, or never eating dessert again, only to scrap the plan a couple of days, if not hours, later. “Unless you’ve had a lot of weight to lose, you don’t understand what it’s like,” she says. “It’s overwhelming, and people look at you like it’s your fault.”

A March 2017 study found that people who internalize weight stigma have a harder time maintaining weight loss. That’s why most experts argue that pushing people toward health goals rather than a number on the scale can yield better results. “When you solely focus on weight, you may give up on changes in your life that would have positive benefits,” says the NIH’s Hall.

It took Jeans five years to lose 75 lb. while on a program at Freedhoff’s institute, but by paying attention to portion sizes, writing down all her meals and eating more frequent, smaller meals throughout the day, she’s kept the weight off for an additional five years. She credits the slow, steady pace for her success. Though she’s never been especially motivated to exercise, she found it helpful to track her food each day, as well as make sure she ate enough filling protein and fiber–without having to rely on bland diet staples like grilled chicken over greens (hold the dressing). “I’m a foodie,” Jeans says. “If you told me I had to eat the same things every day, it would be torture.”

Natalie Casagrande, 31, was on the same program that Jeans was on, but Freedhoff and his colleagues used a different approach with her. Casagrande’s weight had fluctuated throughout her life, and she had attempted dangerous diets like starving herself and exercising constantly for quick weight loss. One time, she even dropped from a size 14 to a size 0 in just a few months. When she signed up for the program, Casagrande weighed 173 lb. At 4 ft. 11 in., that meant she was clinically obese, which means having a body mass index of 30 or more.

Once she started working with the team at the Bariatric Medical Institute, Casagrande also tracked her food, but unlike Jeans, she never enjoyed the process. What she did love was exercise. She found her workouts easy to fit into her schedule, and she found them motivating. By meeting with the clinic’s psychologist, she also learned that she had generalized anxiety, which helped explain her bouts of emotional eating.

It took Casagrande three tries over three years before she finally lost substantial weight. During one of her relapse periods, she gained 10 lb. She tweaked her plan to focus more on cooking and managing her mental health and then tried again. Today she weighs 116 lb. and has maintained that weight for about a year. “It takes a lot of trial and error to figure out what works,” she says. “Not every day is going to be perfect, but I’m here because I pushed through the bad days.”

Freedhoff says learning what variables are most important for each person–be they psychological, logistical, food-based–matters more to him than identifying one diet that works for everyone. “So long as we continue to pigeonhole people into certain diets without considering the individuals, the more likely we are to run into problems,” he says. That’s why a significant portion of his meetings with patients is spent talking about the person’s daily responsibilities, their socioeconomic status, their mental health, their comfort in the kitchen.

“Unfortunately,” he says, “that’s not the norm. The amount of effort needed to understand your patients is more than many doctors put in.”

In an August op-ed published in the journal the Lancet, Freedhoff and Hall jointly called on the scientific community to spend more time figuring out how doctors can help people sustain healthy lifestyles and less on what diet is best for weight loss. “Crowning a diet king because it delivers a clinically meaningless difference in body weight fuels diet hype, not diet help,” they write. “It’s high time we start helping.”

Exactly why weight loss can vary so much for people on the same diet plan still eludes scientists. “It’s the biggest open question in the field,” says the NIH’s Hall. “I wish I knew the answer.”

Some speculate it’s people’s genetics. Over the past several years, researchers have identified nearly 100 genetic markers that appear to be linked to being obese or being overweight, and there’s no doubt genes play an important role in how some people break down calories and store fat. But experts estimate that obesity-related genes account for just 3% of the differences between people’s sizes–and those same genes that predispose people to weight gain existed 30 years ago, and 100 years ago, suggesting that genes alone cannot explain the rapid rise in obesity.

What’s more, a recent study of 9,000 people found that whether a person carried a gene variation associated with weight gain had no influence on his or her ability to lose weight. “We think this is good news,” says study author John Mathers, a professor of human nutrition at Newcastle University. “Carrying the high-risk form of the gene makes you more likely to be a bit heavier, but it shouldn’t prevent you from losing weight.”

Another area that has some scientists excited is the question of how weight gain is linked to chemicals we are exposed to every day–things like the bisphenol A (BPA) found in linings of canned-food containers and cash-register receipts, the flame retardants in sofas and mattresses, the pesticide residues on our food and the phthalates found in plastics and cosmetics. What these chemicals have in common is their ability to mimic human hormones, and some scientists worry they may be wreaking havoc on the delicate endocrine system, driving fat storage.

“The old paradigm was that poor diet and lack of exercise are underpinning obesity, but now we understand that chemical exposures are an important third factor in the origin of the obesity epidemic,” says Dr. Leonardo Trasande, an associate professor of pediatrics, environmental medicine and population health at New York University’s School of Medicine. “Chemicals can disrupt hormones and metabolism, which can contribute to disease and disability.”

Another frontier scientists are exploring is how the microbiome–the trillions of bacteria that live inside and on the surface of the human body–may be influencing how the body metabolizes certain foods. Dr. Eran Elinav and Eran Segal, researchers for the Personalized Nutrition Project at the Weizmann Institute of Science in Israel, believe the variation in diet success may lie in the way people’s microbiomes react to different foods.

In a 2015 study, Segal and Elinav gave 800 men and women devices that measured their blood-sugar levels every five minutes for a one-week period. They filled out questionnaires about their health, provided blood and stool samples and had their microbiomes sequenced. They also used a mobile app to record their food intake, sleep and exercise.

They found that blood-sugar levels varied widely among people after they ate, even when they ate the exact same meal. This suggests that umbrella recommendations for how to eat could be meaningless. “It was a major surprise to us,” says Segal.

The researchers developed an algorithm for each person in the trial using the data they gathered and found that they could accurately predict a person’s blood-sugar response to a given food on the basis of their microbiome. That’s why Elinav and Segal believe the next frontier in weight-loss science lies in the gut; they believe their algorithm could ultimately help doctors prescribe highly specific diets for people according to how they respond to different foods.

Unsurprisingly, there are enterprising businesses trying to cash in on this idea. Online supplement companies already hawk personalized probiotic pills, with testimonials from customers claiming they lost weight taking them.

So far, research to support the probiotic-pill approach to weight loss is scant. Ditto the genetic tests that claim to be able to tell you whether you’re better off on a low-carb diet or a vegan one.

But as science continues to point toward personalization, there’s potential for new weight-loss products to flood the zone, some with more evidence than others.

When people are asked to envision their perfect size, many cite a dream weight loss up to three times as great as what a doctor might recommend. Given how difficult that can be to pull off, it’s no surprise so many people give up trying to lose weight altogether. It’s telling, if a bit of a downer, that in 2017, when Americans have never been heavier, fewer people than ever say they’re trying to lose weight.

But most people do not need to lose quite so much weight to improve their health. Research shows that with just a 10% loss of weight, people will experience noticeable changes in their blood pressure and blood sugar control, lowering their risk for heart disease and Type 2 diabetes–two of the costliest diseases in terms of health care dollars and human life.

For Ottawa’s Jody Jeans, recalibrating her expectations is what helped her finally lose weight in a healthy–and sustainable–way. People may look at her and see someone who could still afford to lose a few pounds, she says, but she’s proud of her current weight, and she is well within the range of what a good doctor would call healthy.

“You have to accept that you’re never going to be a willowy model,” she says. “But I am at a very good weight that I can manage.”