Questions over the colossal investment by US tech companies in artificial intelligence, now running at $400bn a year, continue to come thick and fast.

Will it, sceptics ask, ever be recouped, let alone generate the magical returns AI zealots expect? Leaders of the financial world from Kristalina Georgieva, IMF managing director, to Jamie Dimon of JPMorgan Chase have warned of an abrupt market correction. Could this be one of history's more extreme cases of irrational exuberance?

That phrase, you may recall, was coined by Fed chair Alan Greenspan at the start of the dotcom bubble. He later backtracked, declaring that bubbles could only be detected after the event. The revisionist Greenspan view overlooked that in any bubble there are always shrewd people who see what is coming. For example, on the eve of the 1929 Wall Street crash statistician Roger Babson warned that a "terrific" crash was imminent. More recently Jeremy Grantham, co-founder of US fund manager GMO, famously predicted the bursting of the great Japanese bubble, the dotcom bust and the 2007-08 financial crisis. In the UK fund manager and philanthropist Jonathan Ruffer earned strong returns for his clients around the dotcom blow-up, the great financial crisis and the Covid market plunge. Yet such contrarian voices are always drowned out by those who claim that "this time is different".

Whether today's stock market valuations are irrational is a matter of judgment. But whether investors are behaving irrationally is a different issue. Clearly in all market manias going back to the South Sea Bubble, punters have been intoxicated by stories of untold riches and driven by the fear of missing out (Fomo). Fomo falls short of exuberance but is not exactly irrational. More importantly, in the 21st century when professional investors dominate markets, a paradoxical and perverse rationality is at work among them.

This arises because large amounts of money are delegated by asset owners such as pension funds to asset managers. The job of active managers has traditionally been to maximise returns by assessing fundamental values arising from long-term corporate cash flows. Yet there is a principal-agent problem here. To monitor the managers, asset owners usually benchmark them against an index.

So where managers have a below-index weight in stocks that are rising strongly this puts pressure on them to adopt momentum, or trend following, strategies to improve short-term performance. They thus become late-stage buyers of rising stocks and sellers of falling stocks at the cost of longer-term underperformance. This helps reduce the risk of their being fired, or at least of being fired sooner rather than later.

Research by Paul Woolley and Dimitri Vayanos of the London School of Economics suggests that such momentum trading helps explain the poor performance of active managers and is conducive to a persistent bias to overvaluation. Passive investing, now all-pervasive, is momentum writ large. As well as amplifying mispricing it reduces the liquidity of individual stocks and increases their volatility. Misallocation of capital results, especially where companies use overpriced equity to make acquisitions that encourage industrial and portfolio concentration.

AI brings a further twist to these market dynamics. The Bank of England worries that the use of advanced AI-based trading strategies could lead to firms taking increasingly correlated positions, thereby amplifying shocks. Yet the technology could also hold a key to addressing the resulting instabilities.

Woolley and Vayanos have teamed up with AI experts at Oxford university under Sir Nigel Shadbolt to develop new forms of portfolio analysis designed to disaggregate momentum from fundamental values. This has involved running synthetic portfolios using real price data for periods of up to 30 years.

Initial results reveal managers' skill (or lack of skill) in establishing fundamental value as opposed to their luck in using momentum. In effect, the methodology unpicks the principal-agent conflict. And because this AI diagnostic process provides aggregate data showing how far the market is dominated by momentum, it should help identify bubbles.

The potential here for a big leap in the quality of performance attribution could clearly be valuable for asset owners. But it can never eliminate bubbles. The innate tendency of investors towards exuberance and the corporate compulsion towards leverage will together periodically defy the wisdom of the ages. The snag for shrewd naysayers in any bubble is that the timing of the burst is next to unforecastable.

Could AI help identify skill in fund managers?