Code-switching speech recognition for mixed Hindi–English — dual-pass Whisper fusion, Devanagari→Roman transliteration, word-level language tagging
Browser MediaRecorder (webm/opus) or uploaded file → 16 kHz mono decode server-side.
faster-whisper (CTranslate2, int8) runs twice over the same audio: once forced to English, once to Hindi. Each language prior captures the words the other misses — English pass nails embedded English nouns, Hindi pass nails Devanagari phrases.
Segments from both passes are aligned by timestamp; per segment the hypothesis with the better length-normalized log-probability wins. Hindi output is transliterated to Roman script so the transcript is consistently Hinglish.
Devanagari→Roman transliterator with word-final schwa deletion (कम → kam), spelling-variant canonicalization (bohot/bhaut → bahut), and punctuation cleanup.
Every word is tagged hi/en via lexicon + script heuristics; adjacent tag changes are counted as code-switch points. Drives the colored transcript and the Hindi/English ratio.
Over 100M Indians regularly code-switch between Hindi and English mid-sentence, yet mainstream ASR is monolingual: an English model mangles Hindi words into near-misses, a Hindi model emits Devanagari for everything and loses embedded English terms. This system treats code-switching as the primary case, not an edge case — evaluated on a synthetic Hinglish corpus with known switch points, reporting WER, CER and switch-detection accuracy.