Blog

Meta has announced a groundbreaking AI technology that it calls “the most versatile AI for speech generation.” Dubbed Voicebox, it’s a generative AI model designed for speech generation. However, Meta has decided not to release the model publicly due to potential misuse risks, underscoring the power and potential of this new technology.

Once the tool is released, it will allow several advanced audio editing capabilities, like allowing you to create a voice bot with a very short sample of a real voice. For instance, using an audio sample as short as two seconds long, Voicebox can match the audio style and use it for text-to-speech generation.

It can also recreate a portion of the speech that's interrupted by noise or replace misspoken words without having to re-record an entire speech.

The model is also multilingual, capable of producing speech in six languages: English, French, German, Spanish, Polish, and Portuguese. This versatility is achieved through a new approach called Flow Matching, which allows Voicebox to learn from raw audio and an accompanying transcription. This will make it possible for apps using the technology to provide enhanced real-time translation between languages.

And—since this is the future now—Voicebox can even perform tasks it wasn't specifically trained to do thanks to its in-context learning capabilities.

Why it Matters

This technology could revolutionize the way we create and consume content, especially in the realm of audio and video. Imagine being able to edit audio content as easily as we edit text or images. Or consider the possibilities for multilingual content creation and translation.

“This type of technology could be used in the future to help creators easily edit audio tracks, allow visually impaired people to hear written messages from friends in their voices, and enable people to speak any foreign language in their own voice.”

— Facebook

However, the power of Voicebox also raises important ethical and security concerns. The potential for misuse, such as creating deepfakes or spreading misinformation, is a serious consideration. Meta's decision not to release the model publicly is a testament to these risks. As we move forward, it will be crucial to balance the benefits of this technology with the need for responsible use and regulation.