CyberFreezeDev
Loading
Testing whether a voice agent can handle routine calls in Indian English, Tamil and Hindi.
Live calls need a reply within a second or two; WhatsApp voice notes can take longer. We test the two cases separately.
Can a voice agent handle routine conversations, such as booking an appointment, answering opening hours or qualifying a sales lead, in Indian English, Tamil and Hindi, fast enough to feel natural and with a clean handoff to a person when it should?
We split the problem into two cases because they have very different demands. A live phone call needs a reply within a second or two, or the caller talks over the agent or hangs up. A WhatsApp voice note can take a few seconds to process, because the sender is not waiting on the line. Our WhatsApp CRM automation service work already handles lead capture and routing in text; voice notes are the natural next input.
Findings feed our voice AI agents service. For the terms, see speech-to-text and text-to-speech.
Busy front desks and shop floors miss calls, and missed calls are missed bookings.
A clinic front desk during peak hours, a shop owner on the floor, a sales rep on another call. Missed calls are missed bookings and leads.
Many customers send voice notes in their own language rather than typing.
Plenty of customers would rather send a voice note than type, especially in their own language. A system that only reads text misses them.
Outbound promotional calls fall under TRAI rules, and recording means telling the caller.
Outbound promotional calls in India fall under TRAI's commercial communication regulations, and recording a call means telling the caller. Inbound service calls are simpler, which is one reason we start there.
A cascaded pipeline, so each stage can be swapped and measured on its own.
| Metric | How we measure it | What counts as success |
|---|---|---|
| Word error rate | Transcript compared to a human transcript, reported per language and for code-mixed speech | Low enough that the task fields (name, date, time, phone number) come out right |
| Key field accuracy | Names, dates, times, phone numbers and amounts pulled from the transcript, checked against the answer key | Correct, or the agent asks the caller to confirm |
| Response latency | Time from the end of the caller's speech to the first sound of the reply, split by stage | Short enough that callers do not talk over the agent; measured, not assumed |
| Barge-in | Does the agent stop speaking when the caller interrupts | Stops and listens every time |
| Task completion | Scripted scenarios finished correctly: booking made, question answered, lead qualified | Finished without a person, or handed over correctly |
| Handoff | Does the agent pass to a person when asked, when unsure, or when the caller is upset, with a summary | Never traps a caller who wants a person |
| Pronunciation | Listeners rate how the voice says local names, places and brand terms | A caller understands it first time |
Test recordings use our own voices and scripted scenarios. Real customer calls are not used in published results.
Public research and project documentation first, then our hypotheses. We have not published our own measurements yet.
Speakers usually reply within a fraction of a second, so a three-stage pipeline must keep up.
Conversation research across ten languages found that speakers typically start replying within a fraction of a second of the other person stopping (Stivers and others, PNAS 2009). A three stage pipeline has to fit inside that expectation, or at least fill the gap gracefully.
Whisper large-v2 uses about 4.5 GB in faster-whisper's benchmark, leaving little room on 8 GB.
The faster-whisper project reports about 4.5 GB of peak GPU memory for the Whisper large-v2 model in its own benchmark. That fits on our 8 GB card, but leaves little room for a language model on the same GPU.
Hypothesis: a WhatsApp voice note pipeline will be useful sooner than a live phone agent.
Voice notes remove the latency problem entirely. We expect a reliable voice note pipeline on WhatsApp to be useful sooner than a live phone agent.
Hypothesis: Tamil and English mixed mid-sentence will raise error rates, especially on numbers and names.
Callers in Chennai often mix Tamil and English mid-sentence. We expect error rates on mixed speech to be worse than on either language alone, and numbers and names to suffer most.
Scripted scenarios in each language, by several speakers, with human transcripts as the key.
Scripted booking, enquiry and lead qualification scenarios in each language, by several speakers, with human transcripts as the answer key.
Local and hosted speech-to-text compared on the same recordings, language by language.
Compare local and hosted engines on the same recordings, by language, then write up the method and results as a lab note.
WhatsApp voice notes transcribed, fields extracted, and routed like text leads.
Transcribe, extract fields and route WhatsApp voice notes into the same lead flow as text messages.
Latency measured stage by stage, and the handoff to a person tested before real calls.
Measure end-to-end latency stage by stage and test streaming at each stage, and test the handoff to a person before any agent takes real calls.
Notes will be published in the R&D / Labs notes section of our blog. Recordings use our own voices; no customer calls are published.
| Note | Published when | What it contains |
|---|---|---|
| Method note | Before any scores | The scenarios, languages and speakers in the test set, the transcript rules and how word error rate and key field accuracy are computed |
| Speech-to-text scores | When every engine has run on the full recording set | Word error rate per language and for code-mixed speech, and key field accuracy, with engine, model size, version and hardware |
| Voice note pipeline | After the speech-to-text scores | Field accuracy and processing time for WhatsApp voice notes taken through transcription and routing |
| Live call latency | Only after the handoff tests pass | End-to-end response time split by stage, barge-in results and handoff behaviour |
Multilingual speech models such as Whisper support Tamil, but accuracy varies with accent, audio quality and mixing with English. That is exactly what this track measures before we recommend anything to a client.
They should. We think a voice agent should say so at the start and offer a person at any point, and our test scenarios check for both.
Yes, in principle: the note is transcribed and treated like a text message. Because nobody waits on the line, it is a simpler problem than a live call, and it is the first thing we are building out. See WhatsApp CRM automation service.
Yes. Voiceover is one step in our AI video generation pipelines, and the pronunciation tests here feed into it.
Last updated