The AI Expectation Roller-Coaster

Buckle up, it going to get bumpy

The ‘Trough of Disillusionment’ … is not a topographical feature on the moon nor a nihilist 1970s punk band but rather what faces us as we come to terms with what AI can and cannot do for us.

The Gartner Hype Cycle depicts the lifecycle stages technology passes through from initial development to commercial availability and adoption.

As far as the latest Generative AI is concerned, today we are at, or heading toward, the peak of inflated expectations (of course it is hard to call the peak until we are over it and careening down the other side). The peak is characterized by excitement about and enthusiasm for the technology, and often unrealistic expectations of what it will do.

A recent FT article (‘AI hype has echoes of the telecoms boom and bust’, 14 February 2024) suggests that Sam Altman’s desire to raise $7 trillion (yes, that is trillion with a ‘t’) to reshape the global semiconductor industry to expand its ability to power the next generation of AI is a sure sign that the sector is inhaling its own exhaust - and that we are likely heading for a ‘recalibration’. The FT article points to the telecoms sector’s valuations back when the internet was going to transform the world (before it actually did) – valuations that assumed that the transformation would happen almost overnight. These valuations were unstainable, and the telecoms boom to bust cycle took just 4 years, which was much faster than the internet took to actually change our lives.

“As with the early days of the internet, broader enterprise adoption of AI remains some way off. The transformation triggered by AI may take many years longer than today’s stock prices and funding expectations suggest. Hype and overinvestment are a dangerous combination”

June Yoon, FT ‘AI hype has echoes of the telecoms boom and bust’, 14 February 2024

So the Gartner Hype Cycle would suggest that enthusiasm will rapidly wane as hype meets hard reality, and we will drop into the trough of disillusionment as limits to the technology become clearer and as people question its viability and potential applications.

There are people (and we have come across them presenting at different conferences) who have always argued that this AI thing is just overblown and that worries about the future of work and the need for a paradigm shift in education are over-done. In the depths of the trough, they are likely to take comfort in their own foresight and claim the future will be business as usual.

However as more realistic expectations emerge or as the technology evolves and develops in terms of capability – we will collectively become more knowledgeable about what it can achieve, allowing for more effective and widespread implementation in different scenarios and in different use cases. This is the slope of enlightenment, which finally leads us to the sunlight uplands of the plateau of productivity – where the technology reaches maturity and enjoys widespread market adoption. Higher productivity results as organizations become more proficient at leveraging its capabilities for their benefit.

Roy Amara - American researcher, scientist, futurist, and president of the Institute for the Future – is best known for coining Amara's law which contends that … ‘we tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run’.

To see the Amara’s law (and the Hype Cycle) at work in the real world, we need look no further than the law. The FT’s ‘Innovative Lawyers Europe 2023’ report (published in September 2023) asked senior lawyers about their opinions on Generative AI. Reactions were almost uniformly positive (or grim, depending on your point of view)

“For the first time we have serious disruptive technology […] firms who do not embrace the opportunity AI poses now will not see the erosion going on around them until it is too late” Alastair Morrison, Pinsent Masons

“[Generative AI] is not just a tool, it’s a game-changer”, Lee Ranson, Eversheds

“60% to 70% of the current work in real estate [law] will eventually disappear” Chris De Pury, BCLP

“Generative AI will replace a lot of what lawyers do […] that is revolutionary for lawyers and not just another technological advance” Paula, Gomes Freire, Vieira de Almeida

These views reflect how people felt at the time. Maybe we can call this ‘peak expectation’. We should then expect a steep fall in these expectations over the next few months/years, feeding disillusionment, as Generative AI fails to live up to the hype. We begin to realize that imagined applications have come into contact with reality in terms of what the technology can actually do in its current state. For example … in January 2024 a study from Stanford’s Centre for Human-Centered Artificial Intelligence (HAI) reported its findings on hallucination, testing more than 200,000 legal queries against each of GPT 3.5, Llama 2, and PaLM 2[1]. It found

• Performance deteriorates when dealing with more complex tasks that require a nuanced understanding of legal issues.

• In tasks measuring the precedential relationship between two different cases, most LLMs do no better than random guessing.

• In answering queries about a court’s core ruling, models hallucinate at least 75% of the time.

• Case law from lower courts is subject to more frequent hallucinations than case law from higher courts, showing a tendency to perform better with more prominent cases due to them being more frequently cited and discussed, and thus better represented in training data. 

• Hallucinations are most common among the Supreme Court’s oldest and newest cases, and least common among later 20th century cases, suggesting LLMs’ peak performance may lag several years behind current legal doctrine, failing to internalize case law that is old but still applicable and relevant. 

• Models are susceptible to “contra-factual bias” i.e. the tendency to assume that a factual premise in a query is true, even if it is wrong - likely due to their instruction-following training.

So, with inflated expectations and its revolutionary credentials increasingly under the microscope, in the trough of disillusionment legal AI applications may be struggling under the weight of over-promising and under-delivering.

“Although [Generative AI] is a remarkable development, [most] short-term claims being made about its impact on lawyers and the courts hugely overstate its likely impact” Richard Susskind, technology advisor to UK’s Lord Chief Justice & President of the Society for Computers and Law

It is here that we find out what future impact it will have – expectations are tempered, and the focus moves to addressing weaknesses (which are just a usual part of the development cycle) and refining or improving its application in practical terms. It is here that developers will identify real world added value, and hence onto the slope of enlightenment. It is just a natural progression – technology has to go through the discipline of the trough before we actually reach realistic widely commercialized applications. 

So we would expect legal AI systems to address their tendency to hallucinate (and other drawbacks) coming off the peak of expectations and through the trough of disillusionment. They would demonstrate a more nuanced understanding of legal issues, do a better job at integrating the complete body of caselaw, and identify contra-factual bias.

Amara’s law would suggest that the long-term plateau of productivity actually settles at a higher level of expectation than perhaps projected earlier in the lifecycle of the technology. It just hasn’t happened as quickly as early enthusiasts had hoped, predicted or expected.

“[Generative AI] will mean fundamental changes to the law firm model … eventually” Jeremy Hoyland, Simmons & Simmons

“Lawyers are predisposed to underappreciate the impact or potential impact of technology [and] overvalue things that they are good at” Michael Gerstenzang, Managing Partner of NY headquartered Clearly Gottlieb

Are we collectively over-egging the AI pudding? In the short-term, yes we probably are. Is AI a more profound technology than fire or electricity (as Alphabet CEO Sundar Pichai suggests)? At the moment, no, it probably isn’t, but it may become so. Will AI have a greater long-term impact on our lives and jobs than we currently realize? Amara’s Law suggests it will. Should we be re-tooling our own capabilities to prepare for this future? Absolutely, and the sooner we start the better.

The adoption of AI technologies will be something of a roller-coaster, characterized by discontinuous change, peaks and troughs … but we have to think ahead to how this will change our workplace - standing still is not an option

"The avalanche has already started, it is too late for the pebbles to vote"

J. Michael Straczynski, American filmmaker (Babylon 5)

---

[1] Hallucinating Law: Legal Mistakes with Large Language Models are Pervasive, Jan 11, 2024, Matthew Dahl, Varun Magesh, Mirac Suzgun, Daniel E. Ho