Welcome to Eye on AI! On this version…instructing Deep Ignorance…Cohere’s massive funding and new rent…AI deskilling…Anthropic acquires Humanloop cofounders…ChatGPT market share.
What if stopping AI from serving to somebody construct a organic weapon was so simple as by no means instructing it how?
That query had lengthy intrigued Stella Biderman, govt director of the grassroots nonprofit analysis lab Eleuther AI. In collaboration with the British authorities’s AI Safety Institute, and lead authors Kyle O’Brien and Stephen Casper, Biderman got down to discover the reply — one thing that had by no means been explored in public earlier than.
In a brand new paper, Deep Ignorance, the researchers discovered that filtering dangerous data out of an AI mannequin’s coaching information from the beginning can “bake in” safeguards which can be more durable to tamper with—even in open-source fashions that anybody can obtain and adapt. Crucially, these protections didn’t noticeably harm the mannequin’s general efficiency.
To check the strategy, the workforce skilled variations of an open-source AI mannequin on datasets scrubbed of sure “proxy” data—secure stand-ins for harmful content material, akin to materials associated to bioweapons. The fashions skilled on cleaner information had been much less in a position to produce dangerous data, whereas performing simply as nicely on most different duties.
In an X thread concerning the undertaking, Casper stated the objective was to make LLMs “not solely secure off the shelf, but in addition resist dangerous tampering.” That’s troublesome as a result of most security efforts thus far have targeted on post-training tweaks—adjustments made after a mannequin is constructed. These fixes, akin to fine-tuning a mannequin’s responses to keep away from harmful outputs, can work within the quick time period however are simpler to undo and might typically weaken the mannequin in unintended methods. Pre-training filters intention to bake in security from the beginning, so the mannequin stays secure even when somebody tries to tamper with it later.
Biderman famous that this sort of work is uncommon in public analysis as a result of it’s costly and time-consuming—a barrier for many tutorial and nonprofit teams. Personal AI corporations like OpenAI and Anthropic have the sources, she stated, however keep away from revealing particulars of their pretraining processes for aggressive causes and out of concern over copyright dangers.
“They may completely do that, and who is aware of in the event that they do it,” she stated. “They’re extremely secretive, and don’t actually let you know something.” She pointed to OpenAI’s personal hints that it makes use of some filtering in each its lately launched open-weights mannequin and in its proprietary GPT-4o.
Within the firm’s mannequin card for the open-weights mannequin, OpenAI writes: “To enhance the protection of the mannequin, we filtered the information for dangerous content material in pre-training, particularly round hazardous biosecurity information, by reusing the CBRN pre-training filters from GPT-4o.” In different phrases, the corporate utilized the identical screening course of utilized in GPT-4o to weed out doubtlessly harmful chemical, organic, radiological, and nuclear data earlier than coaching.
For Biderman, Deep Ignorance is supposed to transcend what tech corporations are keen to say publicly. “Having this out in public permits extra individuals to do higher,” she stated. She added that she was motivated partially by the tech trade’s chorus that its large datasets can’t be documented or scrutinized. “There’s a narrative that OpenAI particularly actually likes to inform about how information is unfathomably massive, how might we presumably know what’s in our information,” she stated. “That’s one thing that has pissed me off for a very long time. I feel demonstrating repeatedly that that is mistaken is essential.”
With that, right here’s the remainder of the AI information.
Sharon Goldman
sharon.goldman@fortune.com
@sharongoldman
FORTUNE ON AI
GPT-5’s mannequin router ignited a person backlash towards OpenAI—but it surely could be the way forward for AI – by Sharon Goldman
AI is already making a billionaire increase: There at the moment are 498 AI unicorns—they usually’re value $2.7 trillion – by Julia Coacci
A flood of AI deepfakes challenges the monetary sector, with over 70% of recent enrollments to some corporations being faux – by Lionel Lim
AI IN THE NEWS
Cohere raises $500 million, hires former Meta AI chief Joelle Pineau. Cohere introduced right now that it has raised $500 million in an oversubscribed funding spherical valuing the corporate at $6.8 billion, led by Inovia Capital and Radical Ventures with backing from AMD Ventures, NVIDIA, PSP Investments, Salesforce Ventures, and others. Cohere additionally introduced that it had employed former Meta AI chief Joelle Pineau as chief AI officer and Francois Chadwick as chief monetary officer. “Having Joelle and Francois be a part of similtaneously we’re bringing on this new spherical of funding is known as a game-changer,” Cohere co-founder and CEO Aidan Gomez advised Fortune. “The speed of development in 2025 has been completely unbelievable, with corporations realizing our security-first strategy is essentially distinctive—this supercharges every thing we’re doing.”
AI shortly eroded docs’ means to identify most cancers, research finds. In response to Bloomberg, a brand new research in The Lancet Gastroenterology and Hepatology presents a cautionary story about AI in medication: it may possibly increase efficiency—but in addition trigger ability erosion. Researchers discovered that docs utilizing AI to identify pre-cancerous colon growths grew to become so reliant on the device that, when it was eliminated, their detection charges dropped about 20% beneath pre-AI ranges. The randomized trial, carried out at 4 endoscopy facilities in Poland, suggests over-reliance on AI might make clinicians “much less motivated, much less targeted, and fewer accountable” when working with out it. The findings come as well being methods — together with the UK, which lately funded a significant AI breast most cancers trial — more and more undertake AI to enhance diagnostics.
Anthropic acquires the co-founders and a lot of the workforce past Humanloop. Techcrunch reported that Anthropic has acqui-hired the co-founders and a lot of the workforce behind Humanloop, a UK-based startup identified for its enterprise-focused AI tooling, together with immediate administration, mannequin analysis, and observability. Round a dozen engineers and researchers—together with CEO Raza Habib, CTO Peter Hayes, and CPO Jordan Burgess—will be a part of Anthropic, although the deal didn’t embrace Humanloop’s property or IP. The rent strengthens Anthropic’s enterprise push by including expertise skilled in constructing the infrastructure that helps corporations run secure, dependable AI at scale. Humanloop, based in 2020, has labored with clients like Duolingo, Gusto, and Vanta, and beforehand raised $7.91 million in seed funding from YC and Index Ventures.
AI CALENDAR
Sept. 8-10: Fortune Brainstorm Tech, Park Metropolis, Utah. Apply to attend right here.
Oct. 6-10: World AI Week, Amsterdam
Oct. 21-22: TedAI San Francisco. Apply to attend right here.
Dec. 2-7: NeurIPS, San Diego
Dec. 8-9: Fortune Brainstorm AI San Francisco. Apply to attend right here.
EYE ON AI NUMBERS
78.5%
That’s ChatGPT’s share of the generative AI market right now, in keeping with information by SimilarWeb. The remainder of the sector trails far behind: Gemini (8.7%), DeepSeek (4.1%), Grok (2.5%), Perplexity (1.9%), Claude (1.6%), and Copilot (1.2%).
Lower than three years after its debut in November 2022, ChatGPT is additionally the fifth most-visited web site on the planet—and the fastest-growing, with visitors up 134.9% 12 months over 12 months.
