How do you decide an AI mannequin when it’s already beginning to carry out higher than human beings? That’s the problem confronted by researchers like Russell Wald, govt director of the Stanford Institute for Human-Centered Synthetic Intelligence (HAI).
“As of 2024, there are only a few job classes the place human means surpasses AI, and even in these areas, the efficiency hole between AI and people is shrinking quickly,” Wald mentioned final week in a presentation hosted on the Fortune Brainstorm AI Singapore convention. “AI is exceeding human capabilities and it’s changing into more and more more durable for us to benchmark.”
The HAI releases the AI Index annually, which goals to supply a complete, data-driven snapshot of the place AI is in the present day. At Fortune Brainstorm AI Singapore, Wald shared just a few highlights from the 2025 version of the AI index, such because the growing energy of in the present day’s fashions, the rising dominance of trade on the AI frontier, and the way China is poised to overhaul the U.S.
The next transcript has been flippantly edited for conciseness and readability.
I’m Russell Wald, the manager director of the Stanford Institute for Human-Centered Synthetic Intelligence, or what we name “HAI”.
We’re Stanford College’s globally acknowledged interdisciplinary analysis institute on the forefront of shaping AI improvement for the general public good. HAI was established in 2019 with the objective of advancing AI analysis, training, coverage and follow. And, via our convening function and rigorous research of AI, we have now turn out to be the trusted associate on AI governance for resolution makers in trade, authorities and civil society.
I’m going to speak about what we produce at HAI, which is the AI index, an annual knowledge pushed evaluation of developments in AI that tracks analysis, improvement, deployment and the socio-economic affect of AI throughout academia, authorities and trade.
We see AI efficiency constantly enhance yr over yr. We use Midjourney, a text-to-image generator, asking for a hyper-realistic picture of Harry Potter. And from February 2022 to July 2024, we see quickly growing high quality in these generated photographs.
In 2022, the mannequin produced cartoonish, inaccurate renderings of Harry Potter, however by 2024, it may create startlingly practical depictions. We’ve gone from what mirrors a Picasso portray to an uncanny rendering of Daniel Radcliffe, the actor who performed Harry Potter within the motion pictures.
Due to this constant efficiency progress, we’re more and more challenged with regards to benchmarking these fashions. As of 2024, there are only a few job classes the place human means surpasses AI, and even in these areas, the efficiency hole between AI and people is shrinking quickly. From picture recognition to competition-level arithmetic to PhD-level science questions, AI is exceeding human capabilities and it’s changing into more and more more durable for us to benchmark.
From healthcare to transportation, AI is quickly transferring from the lab to our day by day life. In 2023, the U.S. Meals and Drug Administration permitted 223 AI-enabled medical gadgets, up from simply six in 2015.
On the roads, self-driving vehicles are not experimental. For instance, Waymo, which I repeatedly take whereas residing in San Francisco, is likely one of the largest U.S. operators and supplies over 150,000 autonomous rides every week, whereas Baidu’s reasonably priced Apollo Go robotaxi has a fleet now that serves quite a few cities throughout China.
Enterprise use of AI elevated considerably after stagnating from 2017 to 2023. The newest McKinsey report reveals that 78% of surveyed respondents say their organizations have begun to make use of AI in at the least one enterprise operate, marking a major improve from 55% in 2023.
Pushed by more and more succesful small fashions, the inference price for a system performing on the stage of [GPT 3.5] dropped over 280-fold between November 2022 and October 2024. {Hardware} prices have declined 30% yearly, whereas power effectivity has improved by 40% annually.
Open-weight fashions are additionally closing the hole with closed fashions, lowering the efficiency [gap] from 8% to simply 1.7% on some benchmarks in a single yr. Collectively, these developments are quickly reducing the limitations to superior AI.
Nevertheless, even with inference and {hardware} prices taking place, coaching prices stay out of attain for academia and most small gamers. Practically 90% of notable AI fashions in 2024 got here from trade, which is up from 60% in 2023. And whereas academia stays a prime supply of extremely cited analysis, it does wrestle at this level to remain as superior on the frontier stage.
Mannequin scale continues to develop quickly. Coaching compute doubles each 5 months, datasets each eight, and energy use yearly. But efficiency gaps are shrinking. The rating distinction between the highest and tenth ranked fashions fell from 11.9% to five.4% in a yr, and the highest two fashions are actually separated by simply 0.7%. The frontier is more and more aggressive and more and more crowded.
Lately, AI mannequin efficiency on the frontier has converged, with a number of suppliers now providing extremely succesful fashions. This marks a shift from late 2022, when ChatGPT’s launch, broadly seen as AI’s breakthrough into the general public consciousness, coincided with the panorama dominated by simply two gamers: OpenAI and Google.
One of the vital vital issues to notice is that the transformer mannequin price $930 for Google to coach in 2017—and that’s the T in GPT, the baseline stage of structure—and now in the present day we’re at $200 million to coach Gemini Extremely.
Final yr’s AI index was among the many first publications to spotlight the dearth of normal benchmarks for AI security and accountability evaluations. The index has additionally been analyzing world public opinion. If you’re from a non-Western industrialized nation, you usually tend to view AI positively than not. China has an 83% optimistic view, Indonesia 80%, and Thailand 77%. Whereas Canada is at 40%, the U.S. 39%, and the Netherlands 36%.
I’ll shut with the geopolitical state of affairs. The U.S. nonetheless maintains a lead in AI, adopted carefully by China. Nevertheless, this hole is tightening. My intention is to not exacerbate the concept of an AI arms race between China and the U.S., however as a substitute to spotlight the totally different approaches between essentially the most superior frontier AI mannequin builders.
Over the past a number of years, the U.S. has relied on just a few proprietary mannequin suppliers. In the meantime, China has deeply invested in its expertise base, and extra importantly, an open-source setting. If this development continues, and I seem subsequent yr, at this price, China would surpass the U.S. when it comes to mannequin efficiency.
