CyberFreezeDev- Home
- Blog
- R&D / Labs notes
- Speech to text for Indian languages: Whisper and open models
Speech to text for Indian languages: Whisper and open models
Whisper transcribes most major Indian languages, but published benchmarks show models trained on Indian speech do better. Here is what each open model covers, its licence, and how to handle Tamil or Hindi mixed with English.
Speech to text for Indian languages works today with open models you can run yourself. Whisper transcribes Hindi, Tamil, Telugu, Bengali and most major Indian languages, but published benchmarks show that models trained on Indian speech, such as AI4Bharat's IndicWhisper and IndicConformer, make fewer errors. Code-mixed speech, such as Tamil or Hindi mixed with English, is still the hard case and needs testing on your own audio.
These notes come from our voice AI lab and are for teams who need speech to text for calls, voice notes or meetings in Indian languages.
Which Indian languages does Whisper support?
Whisper's tokenizer lists Hindi, Tamil, Telugu, Malayalam, Kannada, Bengali, Marathi, Gujarati, Punjabi, Urdu, Assamese, Nepali, Sanskrit and Sindhi. It does not list Odia, Konkani, Manipuri, Bodo, Santali, Maithili, Dogri or Kashmiri, so Whisper is not an option for those.
Being on the list does not mean working well. OpenAI's model card says accuracy in a language "is directly correlated with the amount of training data", with lower accuracy on low-resource languages, and that Whisper can hallucinate text nobody said.
Two practical points:
- Set the language. Pass
--language Tamil(or the code) rather than relying on automatic detection. - Use a large model. For Indian languages, start with large-v3. Our guide to self hosted Whisper covers model sizes and GPU memory.
What do published benchmarks show?
The most useful public comparison is Vistaar, from AI4Bharat (Interspeech 2023). It gathers 59 test sets across 12 Indian languages and fine-tunes Whisper on about 10,700 hours of Indian speech to make IndicWhisper, which had the lowest word error rate on 39 of the 59.
Averages on the Hindi test sets, from the Vistaar repository:
| System | Average word error rate (Hindi) |
|---|---|
| Google STT | 23.9% |
| IndicWav2vec | 21.0% |
| Azure STT | 20.0% |
| Nvidia (large) | 18.6% |
| IndicWhisper | 13.6% |
Tamil is harder. IndicWhisper averages 25.3% across the Tamil test sets, against 13.6% for Hindi. On Kathbath, a read speech set, it scores 10.3% for Hindi and 24.2% for Tamil.
Keep two cautions in mind. These are 2023 results on public test sets, and the commercial services have changed since. And a benchmark of clean read speech says little about a noisy phone call to your clinic.
Which open models can you run yourself?
| Model | Languages | Licence | Notes |
|---|---|---|---|
| Whisper large-v3 | 14 Indian languages listed | MIT | One model for all; weakest on low-resource languages |
| IndicWhisper | 12 | MIT | One fine-tuned checkpoint per language |
| IndicConformer 600M | 22 official languages | MIT, gated | CTC or RNNT decoding; 16 kHz audio |
| MMS-1B-all (Meta) | 1,000+ via adapters | CC-BY-NC 4.0 | Non-commercial only |
IndicConformer is the one to test first if you need many languages in one model. Its model card reports a 13.2% word error rate for Hindi on the ARTPARK-IISc Vaani benchmark. Download is gated: you agree to share your Hugging Face details first. You choose CTC or RNNT decoding on each call.
MMS covers the most languages, but its licence rules out a paid product.
How do you handle code-mixed speech?
A caller in Chennai may say a Tamil sentence with an English date and a brand name in the middle. This is normal speech, not an edge case, and it is where systems fail most.
The MUCS 2021 challenge, run on Hindi-English and Bengali-English code-switched speech, reported a baseline word error rate of 32.45% on its code-switching task. That was a 2021 baseline, not a ceiling, but we have not found an open model that publishes Tamil-English code-mixed figures.
Decide early what the transcript should look like. Should "appointment" in a Tamil sentence be written in English letters or in Tamil script? Sarvam's speech API, for example, has a code-mix mode that writes English words in English and Indic words in native script. Pick one convention, write your reference transcripts the same way, and normalise numbers and spellings before you score, or the error rate will mostly measure spelling choices.
Self host or use an API?
Self host when recordings must stay on your network, when volume is high, or when you want to fine-tune on your own callers. Whisper, IndicWhisper and IndicConformer all run on a single GPU server.
One detail for call audio: IndicConformer's example resamples input to 16 kHz, and Whisper works at 16 kHz too. Ordinary phone lines carry narrowband audio, and resampling cannot restore what the line removed, so test on real call recordings, not studio clips.
Use an API when you need many languages quickly and audio may leave your network. Sarvam's API, for example, accepts 22 Indian language codes plus Indian English, takes clips under 30 seconds on its REST endpoint, and offers a batch API with speaker labels for longer files.
Either way, test before you choose:
- Collect 30 to 50 real recordings per language, including code-mixed ones.
- Have a native speaker transcribe them using your chosen script convention.
- Score each candidate on word error rate, and separately on names, dates, numbers and amounts.
- Choose on the fields your workflow needs, not on the headline number.
Testing it on your own calls
Our voice AI lab is recording a test set of Tamil, Hindi, Indian English and code-mixed speech with our own voices; no scores are published yet. Our voice AI agents use hosted speech APIs or open models such as Whisper on your own GPU server, chosen after testing on your callers' accents, names, numbers and addresses. Book a consultation.
