Back in August 2023, Meta revealed an ‘all-in-one’ AI translation model capable of understanding close to 100 different languages.
Dubbed SeamlessM4T (Massively Multilingual and Multimodal Machine Translation), this is Meta’s attempt at creating a ‘universal translator’ akin to the Babel Fish in Douglas Adams’ classic sci-fi series The Hitchhiker’s Guide to the Galaxy.
The team behind the SeamlessM4T tool has now detailed its work in a piece in the journal Nature, revealing the advanced system delivers an all-in-one solution for text-to-text, speech-to-text, speech-to-speech, and text-to-speech translations across an impressive, and growing, array of languages.
Over 400 years of raw audio
SeamlessM4T, which, among other things, is being used to automatically dub videos on Facebook and Instagram, currently supports speech-to-speech translation from 101 to 36 languages, speech-to-text translation for from 101 to 96 languages, text-to-text translation for 96 languages, text-to-speech translation from 96 to 36 languages, and automatic speech recognition for 96 languages. This unified approach overcomes the limitations of traditional cascaded systems, which often require separate subsystems for speech recognition, translation, and text-to-speech synthesis.
By streamlining these processes, Meta says SeamlessM4T outperforms existing models, achieving up to 23% higher BLEU (Bilingual Evaluation Understudy) scores in translation accuracy and demonstrating impressive resilience to background noise and speaker variations.
To create SeamlessM4T, Meta started with 4 million hours (over 400 years) of multilingual raw audio originating from a publicly available repository of crawled web data. The team developed SeamlessAlign, a multimodal corpus containing over 470,000 hours of aligned speech and combined the dataset with cutting-edge machine learning techniques, including SONAR (Sentence-level Multimodal and Language-Agnostic Representations) embeddings, which enable multilingual and modality-agnostic encoding for text and speech.
Meta says that by addressing social and ethical challenges through the use of safeguards, SeamlessM4T can be a valuable tool for global communication. These safeguards reduce gender bias – errors in grammatical gender determination – and mitigate the problem of added toxicity – where offensive words appear in translations but not in the original source.