Charles Koch, the famously private billionaire industrialist, wanted to welcome his dinner guests before they got too far into their meal. It had been a busy day at a seaside summit for 450 conservative donors who support the network of nonprofits, civic groups and political organizations that he and his brother David founded and bankroll. “I’m sure I’ve worn you out,” the 79-year-old said on the broad lawn. Then he reflected on the experiences he suspected he shared with these allies. “We grew up with every advantage,” Koch mused. “Most of you had the same benefits that our parents gave David and me, that is, growing up appreciating and being imbued with the values and skills required for success. If I didn’t have parents like that, I wouldn’t be worth spit. I would be the worst kid on the block … Certain people say I am still.”

Yes, people do say that–and much, much worse–about the Koch brothers. The billionaires help fund a political network that is larger and perhaps more consequential than the Republican National Committee. That machine wields considerable sway over GOP lawmakers and, potentially, the party’s presidential nominee for 2016. Its sprawling influence is just one reason guests ponied up annual checks of at least $100,000 to hear from five White House hopefuls and at least 14 other current or former lawmakers–as well as the two brothers themselves–at this gathering.

These twice-a-year sessions under the banner of Freedom Partners Chamber of Commerce are typically private affairs, held at classy watering holes and spread out over several days. TIME was among a handful of news organizations granted access to this summer’s event, though journalists agreed not to reveal the identities of donors who wanted to remain private. Charles and David Koch are the headline-driving brawn behind this confab for VIP donors, yet thousands of other like-minded conservatives add their cash to the kitty from afar.

Having watched voters send a Democrat to the White House in the past two go-rounds, the Kochs and their allies are recalibrating ahead of 2016. In conversations over snacks, meals and cocktails, there was a grumbling acceptance from the network’s top donors that trying to keep earlier events secret had backfired. “The Koch brothers could be depicted as comic-book villains,” says Craig Snider, the 59-year-old son of the family that owns the Philadelphia Flyers. “They are a private family. They never really wanted the attention.” As he sipped chilled white wine in one of the St. Regis Monarch Beach’s courtyards, Snider shook his head at the overwhelmingly negative coverage of the Kochs and their partners. “Our side has done a very bad job telling our story. We’ve been defined by the other side.”

So what is it like to observe the mysterious Koch brothers up close? It’s not all that different from watching two admired grandfathers oversee a large family reunion. They weave through the retreats they convene with an unassuming style that, were it not for the security trailing them, would be like that of any other septuagenarians, moving at a slower pace but refusing to be sidelined. Charles, talkative and engaging, lives in Kansas and has lost little of the quick, dry humor he used to tremendous success in business negotiations. David, somewhat quieter by nature, enjoys a more cosmopolitan life in Manhattan, where the New York City Ballet’s performance hall at Lincoln Center carries his name.

They were born in Wichita, Kans., in the years leading up to the U.S. entrance into World War II. Their upbringing reflected their father’s hard-nosed approach to life: disagreements were settled by fistfights. Fred Koch owned a sprawling Midwest industrial giant yet required his children to learn the trade rather than enjoy a gilded life. “I got my butt kicked every day,” Charles recalls. “Father had me work every minute from the time I was 6.” Both brothers went to MIT and earned graduate degrees in engineering before returning to Kansas, where they expanded the company their father founded into what today has become the second largest privately held company in the U.S.

The Kochs are often described as either ultra-conservative or libertarian, but those labels don’t fully explain their ideology. Yes, they believe that government has become too big; they fiercely oppose mandates and regulations, and they could not be more horrified by what they call the permanent Washington establishment. And it is fair to say they don’t care for President Obama. If their wish list of government rollbacks were achieved, it would help the bottom line for Koch Industries, a vast collection of companies and interests that produce everything from Brawny paper towels to Stainmaster Carpets to as many as 600,000 barrels of crude each day.

But some of what this network is trying to accomplish at sessions like those held here is at odds with Koch Industries’ bottom line. The groups oppose government subsidies of all kinds, even those that help the Koch companies’ profits. They would like to see Congress kill the Export-Import Bank and the ethanol subsidies that benefit the family operations that turn Iowa corn into fuel. Koch-backed groups have made building the Keystone XL pipeline a must-do task, even though it would compete with Koch Industries’ refineries. “The prevailing view created by the mainstream media is that this is to enrich Charles and David Koch,” says Mark Holden, general counsel for Koch Industries and one of the brothers’ top lieutenants. “We take a lot of positions that are better long-term for all of us in the country, even though in the short term we would lose money.”

All of these arguments were raised during the summit in Dana Point, which has become something of a refresher course on conservative thinking. For instance, guests attended one session to hear how Chile reduced its poverty rate from 50% to 8% in a generation, but at a political cost. Other donors received updates on the Koch-led crusade against mandatory minimum sentences for nonviolent drug offenders as fiscally, constitutionally and morally unacceptable. Some guests attended a small dinner to talk about free-speech rights on college campuses with Mitch Daniels, a former White House budget chief and Indiana governor who now serves as president of Purdue University. But economic policy, really, ran through most of the discussions.

Charles Koch told his allies, including CEOs of well-known American companies, to ditch government tax breaks and subsidies for their own good. “Obviously, this prescription will not be an easy pill for many businesspeople to swallow,” he said, the sun setting over the Pacific Ocean, just a golf-cart ride away. “Because short-term, taking the principled path is going to cost some companies some profits, as it will for Koch Industries.”

Whatever their agenda, losing isn’t part of it. In all, groups under the Koch umbrella plan to spend about $889 million before Election Day 2016, and roughly two-thirds of it will try to determine how voters cast their ballots. Part of their advantage is in how they charter themselves: the groups can accept unlimited donations, and because of the way many are structured, donors’ identities can largely be kept secret. (By contrast, the RNC has fundraising caps and must disclose everyone who gives $200 or more.)

The Koch-based network now is looking at how best to spend the money. During the summit, top Republican strategists told the Koch faithful that four states would be the biggest focus for the next two years: Florida, Ohio, North Carolina and Virginia. It is almost impossible for an eventual Republican nominee to win the White House without those states. The donors are also continuing to invest in i360, a Koch-built database containing some of the most sophisticated information on voters’ interests and habits. “You can have all the academic debate you want to,” says Art Pope, a mega-donor from North Carolina and Koch friend. “But eventually it takes elected leaders to change the laws and change the policies.”

That focus helps explain why Marco Rubio, Scott Walker, Ted Cruz, Jeb Bush and Carly Fiorina all spent time with these donors at this summit. Each took turns praising the Koch network’s vision of more-limited government while lending their voices to the chorus of praise for what Charles and David Koch have accomplished. Rubio, Walker and Cruz are favorites with this crowd and old hands at the weekend. Bush was attending his first summit and soothed some donors’ unease about his brother; these summits were born out of frustration that federal spending was growing under President George W. Bush. Few knew Fiorina, and she worked the crowd hard to fix that.

But something else was visible at the Dana Point gathering as well. Charles Koch recognizes that the GOP cannot win a national election if it cannot expand its appeal beyond the types of conservatives who huddle with him at these retreats. Just consider the organizational chart of groups that now operate with Freedom Partners’ backing: grassroots-driven Americans for Prosperity, youth-focused Generation Opportunity, Hispanic-oriented LIBRE Initiative, the female-directed Concerned Women for America. At the same time, millions of dollars are flowing through the Koch network to the United Negro College Fund; its president, Michael Lomax, lectured these donors on why historically black colleges and universities matter during an outdoor dinner party at this summit.

On the lobbying front, Freedom Partners–backed groups have linked arms with the liberal Center for American Progress and ACLU for a bipartisan push on criminal-justice reform. Such work has won the Kochs notice–and some guarded praise–from the White House. No one expects the entente to last long. “Last summer, some of them were attacking us. Now we’re working with them,” says Holden, the Koch Industries lawyer. “I know they’re going to be attacking us later.”

The criticism is unlikely to end no matter how wide this moneyed network throws open its doors. But the modest amount of transparency suggests that the Koch brothers are starting to contemplate their legacies. For David Koch, it will be philanthropic giving that is almost unrivaled: $1.3 billion to charities, including $225 million to Memorial Sloan Kettering Cancer Center in New York City.

For Charles Koch, the goal is a realignment of policy and politics that, in his view, will preserve the America he knew as a child, when a kid from Kansas could turn the family business into a global player. “These guys are using business principles to create a political solution,” says Tim Busch, an Orange County lawyer and loyal Koch donor. “They’re creating a force to be reckoned with, so that the political parties have to deal with them and respect them.”

Write to Philip Elliott at philip.elliott@time.com.

This appears in the August 17, 2015 issue of TIME.

SPONSORED FINANCIAL CONTENT

Read More From TIME

EDIT POST