Updated: September 10, 2014
With the S&P 500 having recently crossed into record territory, many market watchers worry that stocks are overpriced and headed for a fall. You might think such talk would slowly tamp down investor enthusiasm. In fact, there's evidence that the opposite is more often true: The higher prices go, the more market players -- amateur and professional alike -- expect them to rise.
Why? To answer that, it helps to understand the worriers' case. Much of the discussion is focused on the Shiller P/E ratio. Named for Nobel Prize winning economist and Irrational Exuberance author Robert Shiller, the ratio measures the price of stocks relative to the average of the past 10 years earnings. When the Shiller P/E is high, returns over the following decade tend to stink. The Shiller P/E isn't much of a short-term market timing tool — the number can remain high for years — but the long term correlation looks pretty convincing.
Right now, the Shiller P/E is around 26. Look at the chart above and you'll see that buyers at that price scratched out moderate returns at best. The number isn't screaming "crash," but it hardly makes stocks look like a bargain, either.
The lesson here ought to be obvious enough. If you pay a lot for something, you stand to earn less on it. You'll profit more—or lose less—on a house you buy for $100,000 than you would on the same house at $140,000. To see more clearly how that works with stocks, just flip the P/E ratio of 25 over (that is, divide the E by the P) and you'll see that earnings are in theory paying investors a "yield" of 3.8%. In 2009, when the Shiller P/E was as low as 15, the earnings yield was 6%.
So what are investors thinking when they pay high prices for stocks? Don't they know they are likely to get lower returns?
A lot of economists assume that, yes, buyers do know that. And that they are fine with it. Perhaps attitudes toward risk have changed, they figure. One day people thought they needed to earn, say, five percentage points above the return on bonds to justify the risk of owning stocks. But maybe now two or three or four points will do just fine. That's not such an odd idea. As economist John Cochrane explains (citing Eugene Fama, who shared the Nobel with Shiller), risk appetites could go up, quite rationally, when the economy is strong. He writes: "Is it not plausible that people say 'yeah, stocks aren't paying a lot more than bonds. But what else can I do with the money? My business is going well. I can take the risk now.'"
By this way of thinking, back in 2000, when the Shiller P/E hit 40, the "expected return" required was low. But on Planet You, Me and Everyone You've Ever Talked With About Stocks, the late-1990s-to-2000 bull market was a time when people expected enormous returns. Many investors were thinking that settling for less than 15% returns was for chumps!
A recently published article by Robin Greenwood and Andrei Shleifer of Harvard, "Expectations of Returns and Expected Returns," looks at the evidence from multiple surveys of investors and concludes that they do indeed expect higher returns when prices are already high. (The authors measured stock prices differently than Shiller, but the idea is roughly the same.) And why is this? Because, the authors suggest, investors are simply projecting recent gains forward into the future.
And it's not just your day-trading brother-in-law who does this kind of naive extrapolation of the past into the future. Greenwood and Shleifer write: "A substantial share of investors, including individuals, CFOs, and professional investors hold extrapolative expectations about returns. When stock prices are high, and when they have been rising, investors are optimistic about future market returns."
If some market participants are irrational, then why aren't smarter traders coming in to take advantage of that, and in the process pushing prices back down? In another paper, the economists and two other coauthors suggest that "rational" traders figure out that others are overreacting to the trend, and so they decide to ride the wave for a while.
Until when? Maybe until the economic news turns bad -- at which point investors might overreact in the other direction. Good luck timing those turns. (Read: Don't try.) As economist and blogger Noah Smith points out, Greenwood and Shleifer's work doesn't help you time the market. But it does add up to a good case for lowering your own expectations.