This Is Warren Buffett’s Biggest Weakness According to Warren Buffett

This post is in partnership with Fortune, which offers the latest business and finance news. Read the article below originally published at Fortune.com.

Can you build one of the world’s most successful companies and also be a great boss? Warren Buffett may be one of the few executives to accomplish this feat.

In acquiring the dozens of companies that today make up his $315 billion Berkshire Hathaway, Buffett has never dirtied his hands much with the sort of cost-cutting, layoffs, and management shakeups that often follow acquisitions at other firms. While Buffett’s style of allowing even underperforming businesses to languish is part of what makes him so unique, it is also his greatest shortcoming, the Berkshire CEO and chairman admitted during the company’s annual shareholder meeting this past weekend in Omaha.

MORE: 6 things I learned at Buffett’s annual meeting

Asked by a shareholder to highlight his weaknesses, Buffett said that as a manager, he can be “sloppy.” “A pure weak point is, I’m slow to make personnel changes,” he said on Saturday. “There will be times when what you might call our lack of supervision of our subsidiaries means we’ll miss something.”

Without getting into specifics, he recalled one example. “We had a guy, we loved the guy, and it wasn’t killing us in our business, and how long before we went to somebody else?” (Neither he nor Berkshire’s vice chairman Charlie Munger, who sat beside Buffett on stage, could remember.) “We’ve waited too long on one manager sometimes,” Buffett said.

Buffett’s comments came amid grumbling from some investors gathered in Omaha that the Oracle is too hands-off and perhaps should take a page from activist shareholders like Pershing Square Capital’s Bill Ackman or turnaround specialists like private equity firm 3G Capital. Indeed, several questions from shareholders and analysts at Saturday’s meeting focused on whether Berkshire, which teamed up with 3G in 2013 to purchase ketchup maker Heinz (where management was then swiftly replaced), would adopt 3G’s “hands-on” approach for future acquisitions.

MORE: Buffett: We’d like to own more of Heinz

While Buffett praised 3G as a partner (“They’re smart, they’re focused; they’re very hard-working and determined.”) he also noted that “they’re never satisfied,” and defended Berkshire’s gentler way of doing things. For one, Buffett doesn’t like telling his subsidiaries that they need to get rid of employees or restrict their actions in other ways. “We’re not big disciplinarians,” Buffett said. As Munger put it, “By the standards of the rest of the world, we over-trust.”

So much does Buffett trust the CEOs of Berkshire companies that he doesn’t keep close track of how much money they send him and when, or bother to double-check that a company even exists. He’s never visited Forest River, the Indiana-based recreational vehicle manufacturer that Berkshire owns, for example. “I hope it’s there,” Buffett joked. “I could see them saying, ‘What figure should we send Warren this month?'” he said, laughing. (“I don’t want to encourage managers of subsidiaries to do a different way of behaving, if they’re listening,” he added.)

When you leave companies alone to run themselves, sometimes things go wrong, Buffett acknowledged. But he took issue with critics who complain that Berkshire’s returns would be higher if Buffett ran a tighter ship. “What they won’t be able to measure is how much on the positive side we have achieved with so many other people because we gave them that leeway,” he said, noting that Berkshire doesn’t have in-house general counsel or human resources.

As for Ackman’s brand of activism, Buffett seemed almost offended by any likeness between Berkshire and the hedge fund manager. First, unlike Ackman, who has lately made headlines for his short position in Herbalife and hostile bid for Botox manufacturer Allergan, Buffett doesn’t short stocks or trade in derivatives. He’s also skeptical that activists’ recent success is sustainable, saying they’re usually looking to cause a short-term swing in a stock price, not permanent changes to improve a business. Munger, for his part, swore off investor activism in general, saying, “I don’t think it’s good for America.”

MORE: Warren Buffett, activist investor?

Will Buffett’s successor be the hard-line boss that Buffett isn’t? Don’t count on it.

One of Berkshire’s most valuable assets, Buffett and other investors reiterated over the weekend, is other businesses’ willingness to sell to Berkshire — often at lower prices than what private equity firms would pay — because they trust Buffett to mind the store. As hedge fund manager Whitney Tilson of Kase Capital Management put it on Friday, “Berkshire is like a museum, like the Met, where people will sell their most valuable art just because they know it’s going to be safe there.”

Added Buffett, “We keep promises, and that means we sometimes end up holding on to some businesses that haven’t lived up to expectations.”