In June, Bellingcat ran 500 geolocation checks, evaluating LLMs from varied corporations towards one another, in addition to Google Lens – a staple software for locating the situation of photographs.
On the time, ChatGPT o4-mini-high emerged because the clear winner, with Google Lens outperforming most different fashions. Simply two months later, with new variations of those AI instruments obtainable, we re-ran the trial – this time together with Google “AI Mode,” GPT-5, GPT-5 Pondering, and Grok 4 into the combo.
The unique check used 25 of Bellingcat’s personal vacation photographs. From cities to distant countryside, the pictures included scenes each with and with out recognisable options – corresponding to roads, signage, mountains, or structure. Pictures had been sourced from each continent.
For the up to date trial, 5 check photographs had been excluded, as they’d appeared in a earlier article, thus compromising the integrity of the outcomes.
All 24 fashions’ responses had been ranked on a scale from 0 to 10, with 10 indicating an correct and particular identification (corresponding to a neighbourhood, path, or landmark) and 0 indicating no try to establish the situation in any respect.
Google AI Mode was proven to be probably the most succesful geolocation software general.
Grok 4 gave each higher and worse solutions in comparison with Grok 3 however, on common, scored marginally larger. Nonetheless, it was nonetheless much less correct than older variations of Gemini and GPT.
GPT-5, even in ‘Pondering’ and ‘Professional’ modes, was a substantial downgrade compared with the capabilities demonstrated by GPT o4-mini-high. In a single instance, of a metropolis avenue with skyscrapers within the background, o4-mini-high accurately recognized the road, whereas GPT-5 in Pondering mode pointed to the fallacious nation.
Help Bellingcat
Your donations straight contribute to our capacity to publish groundbreaking investigations and uncover wrongdoing world wide.
Regardless of delivering quicker solutions, GPT-5 appeared to sacrifice accuracy. A stunning variety of errors and a common sense of disappointment within the new mannequin have additionally been reported by different customers.
Bellingcat examined GPT-5 and its ‘Pondering’ mode through the Plus subscription, which prices roughly the identical as entry to 04-mini-high previous to its retirement. 5 of probably the most troublesome check photos had been additionally run by means of GPT-5 Professional. However even Professional, with a premium price ticket of €200 monthly, did not geolocate the photographs any extra precisely than GPT 04-mini-high.
A Seaside, a Lodge and a Ferris Wheel
The disparity between Google and the GPT fashions turned much more obvious in Take a look at 25 – a photograph of a shoreline lodge in Noordwijk, the Netherlands, with a Ferris wheel rising simply past the dunes.
Within the earlier trial, most older fashions – together with these from GPT, Claude, Gemini and Grok – precisely recognized the nation because the Netherlands however did not find the city. Many latched onto the Ferris wheel however pointed as an alternative to the seaside city of Scheveningen, which additionally has a Ferris wheel, although located on a pier, not among the many sand dunes.
Nonetheless, the latest fashions, GPT-5 Professional and Pondering, had been even much less correct, figuring out a seashore in France – a completely completely different nation.
Sadly for open supply researchers, following the discharge of GPT-5, OpenAI eliminated the choice to pick older fashions corresponding to o4-mini-high. After a wave of unfavorable suggestions, OpenAI reinstated GPT-4o because the default mannequin for paid subscribers. Nonetheless, probably the most succesful geolocation fashions recognized in Bellingcat’s testing stay inaccessible.
Google AI Mode, however, was the primary, and solely mannequin to date, to accurately establish Noordwijk as the situation in Take a look at 25.
Although AI Mode is powered by a model of Gemini 2.5, it outperformed Gemini 2.5 Professional Deep Analysis in these checks. Described by Google as its “strongest AI search, with extra superior reasoning and multimodality,” AI Mode geolocated check photos with higher accuracy than any GPT fashions, together with our earlier winner, o4-mini-high.
AI Mode is at present solely obtainable in India, United Kingdom and the USA.
The vast majority of fashions, sooner or later, returned a hallucination. Customers mustn’t rely solely on the solutions supplied by LLMs. Even the very best choices, together with Google AI Mode, nonetheless, at occasions, confidently level to the fallacious location.
The distinction in fashions’ capabilities in contrast with simply two months in the past exhibits how shortly this discipline is evolving. Nonetheless, OpenAI’s latest adjustments additionally recommend that progress shouldn’t be assured, and that AI’s capacity to geolocate might plateau and even worsen over time. As new fashions emerge, Bellingcat will proceed to check them.
Because of Nathan Patin for contributing to the unique benchmark checks.
Bellingcat is a non-profit and the power to hold out our work relies on the sort help of particular person donors. If you want to help our work, you are able to do so right here. You may as well subscribe to our Patreon channel right here. Subscribe to our E-newsletter and observe us on Bluesky right here and Instagram right here.
