Warren Buffett is the most studied investor of all time. And with a couple of exceptions, it’s hard to find anyone who has studied him as much as Robert Hagstrom. Hagstrom recently appeared on the We Study Billionaires podcast and spoke at length about one of the investing world’s great mysteries.

That mystery is as follows:

Warren Buffett is widely regarded as the most successful investor ever. Rather than keep the secret sauce locked away, Buffett and his late business partner Charlie Munger have shared their knowledge liberally for decades. What’s more, the main principles they preach are rather simple – ignore market folly, buy shares in great companies and let compounding work its magic. The financial rewards of replicating the methods and success of these two men – even 1% of it – would be enormous. And yet despite all of the above, very few investment managers have pulled it off.

Why not? According to Hagstrom, it boils down to psychological barriers as much as anything else.

Barrier 1: The unbearable pain of losing to win

Buffett is the archetype of what Hagstrom describes as a ‘high active share’ investor. High active share managers are defined by holding portfolios that are very different to the big equity benchmarks. They are often highly concentrated, and this combination naturally leads to big differences in performance versus the benchmark.

Done well, this can bring about a truly impressive track record. The nature of investing this way, though, means that being smart enough to have good investment ideas isn’t enough. You also need to be comfortable with losing a lot of the time.

“We looked at the high active share managers in the Warren Buffett Way [his book]. Phenomenal long term track records, but their batting average was about 50%. They outperformed month to month, quarter to quarter, year to year, only 50 percent of the time. The other 50 percent of the time they underperformed.”

I recently read some lecture notes from Joel Greenblatt’s value investing classes at Columbia and this theme came up again and again. According to Greenblatt, value investing works on average because it doesn’t work all of the time – if it did, everybody would do it and it wouldn’t work. It also reminded me of the classic Peter Lynch quote about your stomach being more important than your brain.

We hear quotes like this and we nod along. Yet most of us are hard-wired against living up to them.

Hagstrom cites Daniel Kahneman and Amos Tversky’s Prospect Theory as the main reason for this. Kahneman and Tversky found that investors feel twice as much pain from losses as they feel joy from equally big gains. For most people, this makes a high active share approach like Buffett’s difficult to stomach, even if there is clear evidence of its ability to deliver outstanding relative returns.

I imagine this would be even harder as a fund manager. Not only would you have to deal with your own emotions, but you’d also have to deal with those of your clients and colleagues, too. Buffett has alluded to this advantage over fund managers many times. At Berkshire, he is not investing funds at risk of being withdrawn by skittish clients. He is investing permanent capital and enjoys the support of a shareholder base that would follow him off a cliff. Of course, it helps that one of those shareholders – with 31% voting power no less – is Buffett himself.

Barrier 2: The power of self-interest

Another reason there aren’t more Buffetts out there? Other approaches to investing are deeply entrenched and protected by webs of self-interest.

The big one Hagstrom takes aim at is Modern Portfolio Theory (MPT), which is diametrically opposed to the bumpy returns and concentration of a ‘high active share’ approach. By contrast, MPT views stock price volatility as the very definition of risk and seeks to eliminate it through diversification.

According to Hagstrom, MPT took hold in the 1970s amid a lack of strong voices championing other investment approaches and definitions of risk. This happened to coincide with a near 30-year secular bull market starting in the US, meaning that prestigious academic careers and huge amounts of assets under management became entwined with this approach.

“Go tell guys that have billions of dollars in modern portfolio theory. Oh, you know that money management practice that is making you millions of dollars a year? That gets you all the luxuries and everything that you want? Oh, you need to shut that down. It doesn’t make any sense anymore. No, they’re not going to do that. They’re going to defend that till hell freezes over.”

Even leaving MPT aside, there is little incentive for institutions with huge assets under management to take a highly active approach. At a certain point, the game becomes more about protecting assets under management than swinging for outperformance. In that situation, loss aversion kicks in again and ‘closet indexing’ becomes far more attractive than a Buffett style approach.

Barrier 3: The difficulty of focusing on what matters

According to Hagstrom, what really makes Warren Buffett different from other investors is what he does not spend time thinking about.

“[Buffett] doesn’t think in terms of common stocks, sectors, correlations, diversification. He doesn’t think about stock market theories. He doesn’t think about macroeconomic concepts. He just thinks about the business. Now, compare and contrast that with an institutional money manager…

The majority of people spend 90% [of their time] pontificating about the market, the economy, geopolitics, the presidential election. Who cares?”

Unfortunately, most investment managers need to think about those things. Why? Because that’s what their clients are thinking about. As Hagstrom put it, “9 out of 10 phone calls from clients are going to be asking these questions. You’re not going to have a long career if you don’t at least contribute something.”

For some reason, I found the image of Buffett constantly being torn away from Apple’s 10-K to answer phone calls asking him about the election quite amusing. But it raises a serious question. Were it not for the unique structure he built for himself at Berkshire, would Buffett also have failed to invest the way he wanted to?

More on Buffett