Best Overall Model: 🏆 smallestai_streaming delivers the lowest average Word Error Rate (WER) across English, Hindi, and code-switched audio, while also handling noisy and disfluent speech effectively. It edges out deepgram_nova3_streaming in amost every category, making it the top choice for multilingual and mixed-context scenarios.Note: AssemblyAI’s streaming ASR does not support Hindi or code-switched transcription, it is English-only in streaming mode.
The internal test dataset comprises a diverse set of speech samples designed to evaluate ASR performance across real-world conditions, including language variation, background noise, and natural conversation artifacts.
Code-Switched: Hindi-English mix within the same conversation.
Example: “यह movie बहुत अच्छी थी but ending थोड़ी confusing लगी”
How it was created: Generated using our in-house TTS on curated code-switched text, ensuring natural alternation between languages in the same sentence or utterance.
Hindi: Traditional Hindi in Devanagari.
Example: “हमने उसका जन्मदिन मनाया”
How it was created: Recorded and synthesized from native Hindi speakers using varied topics, from casual conversations to descriptive narratives.
English: Standard English in Latin script.
Example: “jovial joggers joyfully joined jogging jaunts justifying joyful jolliness”
How it was created: Includes tongue twisters, technical/scientific terminology, and diverse domains such as technology, healthcare, and finance to evaluate robustness across vocabularies.
Disfluency: Audio containing hesitation words, repetitions, and self-corrections.
Example: “see uh uh i i when i went i thought the food was not good”
How it was created: Real recordings of people speaking naturally, sourced from Atoms call recordings to capture genuine in-the-wild speech patterns.
Noisy: Audio with background interference, low quality mics, or multiple speakers.
How it was created: Real recordings from Atoms call data with actual customer interactions, ambient sounds, and overlapping speech, replicating real-world ASR deployment conditions.
