OpenAI expands Realtime API with new voices and cuts costs for builders

OpenAI up to date its Realtime API right now, which is at present in beta. This replace provides new voices for speech-to-speech functions to its platform and cuts prices related to caching prompts. 

Beta customers of the Realtime API will now have 5 new voices they will use to construct their functions. OpenAI showcased three of the brand new voices, Ash, Verse and the British-sounding Ballad, in a publish on X. 

Two Realtime API updates:

– Now you can construct speech-to-speech experiences with 5 new voices—that are way more expressive and steerable. ???

– We’re reducing the value through the use of immediate caching. Cached textual content inputs are discounted 50% and cached audio inputs are discounted… pic.twitter.com/jLzZDBrR7l

— OpenAI Builders (@OpenAIDevs) October 30, 2024

The corporate stated in its API documentation that the native speech-to-speech function “skip[s] an intermediate textual content format means low latency and nuanced output,” whereas the voices are simpler to steer and extra expressive than its earlier voices. 

Nonetheless, OpenAI warns it can’t supply client-side authentication for the API now because it’s nonetheless in beta. It additionally stated that there could also be points with processing real-time audio. 

“Community situations closely have an effect on real-time audio, and delivering audio reliably from a consumer to a server at scale is difficult when community situations are unpredictable,” the corporate shared.

OpenAI’s historical past with AI-powered speech and voices has been controversial. In March, it launched Voice Engine, a voice cloning platform to rival ElevenLabs, nevertheless it restricted entry to just a few researchers. In Could, after the corporate demoed its GPT-4o and Voice Mode, it paused utilizing one of many voices, Sky, after the actress Scarlett Johansson spoke out about its similarity to her voice. 

The firm rolled out ChatGPT Superior Voice Mode for paying subscribers (these utilizing ChatGPT Plus, Enterprise, Groups and Edu) within the U.S. in September. 

Speech-to-speech AI would ideally let enterprises construct extra real-time responses utilizing a voice. Suppose a buyer calls an organization’s customer support platform. In that case, the speech-to-speech functionality can take the individual’s voice, perceive what they’re asking, and reply utilizing an AI-generated voice with decrease latency. Speech-to-speech additionally lets customers generate voice-overs, with a consumer talking their traces, however the voice output just isn’t theirs. One platform that provides that is Reproduction and, in fact, ElevenLabs.  

OpenAI launched the Realtime API this month throughout its Dev Day. The API goals to hurry up the constructing of voice assistants.

Reducing prices

Utilizing speech-to-speech options, although, might get costly. 

When Realtime API launched, the pricing construction was at $0.06 per minute of audio enter and $0.24 per audio output, which isn’t low cost. Nonetheless, the corporate plans to decrease real-time API costs with immediate caching. 

Cached textual content inputs will drop by 50%, and cached audio inputs shall be discounted by 80%.

OpenAI additionally introduced Immediate Caching throughout Dev Day and would preserve often requested contexts and prompts within the mannequin’s reminiscence. This may drop the variety of tokens it must create to generate responses. Reducing enter costs, might encourage extra builders to connect with the API. 

OpenAI just isn’t the one firm to roll out Immediate Caching. Anthropic launched immediate caching for Claude 3.5 Sonnet in August.