Self hosted Whisper: transcribe calls on your own server

Yes, you can run Whisper on your own server: OpenAI released the code and model weights under the MIT licence, and faster-whisper runs the same models faster and with less memory. A single GPU with 8 GB, or even a modern CPU for smaller models, can transcribe call recordings without sending a second of audio to a cloud API. Speaker labels need a separate diarisation model, and accuracy with strong accents or mixed languages needs testing on your own calls.

This guide is for the technical lead at a clinic, a contact centre, a law firm or a sales team who wants speech to text without the recordings leaving the building.

Whisper or faster-whisper?

Whisper is OpenAI's reference implementation in Python and PyTorch. It needs ffmpeg and is the easiest place to start.

faster-whisper reimplements Whisper on the CTranslate2 engine. Its README says it is up to four times faster than the original for the same accuracy, with less memory, and supports 8-bit quantisation on CPU and GPU. It also has a voice activity filter that skips silence, word level timestamps and a batched pipeline. It is MIT licensed too, and needs CUDA 12 and cuDNN 9 for GPU use.

For a server that processes recordings in a queue, faster-whisper is the usual choice. whisper.cpp is a third option for machines without a supported GPU.

Which model size should you use?

From the Whisper README. Speeds are relative to large, measured by OpenAI on an A100 in English.

For business calls, start with turbo or large-v3 and drop to a smaller model only if speed forces it. If you need English translations of Arabic or Hindi calls, turbo will not do it; the README says to use medium or large for translation.

GPU or CPU?

faster-whisper's maintainers publish a benchmark on 13 minutes of audio. With the large-v2 model on an RTX 3070 Ti 8 GB, the same GPU model as our lab laptop, they report:

• 16-bit: 1 minute 3 seconds, 4,525 MB of VRAM
• 8-bit: 59 seconds, 2,926 MB
• 8-bit, batch of 8: 16 seconds, 4,500 MB

With the small model on an Intel Core i7-12700K using 8 threads, 8-bit took 1 minute 42 seconds, or 51 seconds batched.

These are their figures, not ours. They suggest a consumer GPU handles a large model comfortably, and a CPU server is enough for the small model when calls can wait in a queue overnight.

How do you tell who said what?

Whisper produces text, not speaker labels. Three ways to get them:

1. Record each side on its own channel. If your phone system can save the agent and the caller as separate left and right channels, use it. Transcribe each channel separately and you know who spoke without any diarisation model.
2. WhisperX. It adds word alignment and speaker labels from pyannote's speaker-diarization-community-1 model. That model is CC-BY-4.0, gated on Hugging Face (you accept its terms and use a token to download), and can then run offline from a local copy, on CPU or GPU. You can tell it the exact number of speakers, which helps on two-party calls.
3. whisper-diarize, listed by faster-whisper, which uses NVIDIA NeMo for the speaker step.

Set expectations. WhisperX's own README says overlapping speech is not handled well and diarisation is far from perfect. pyannote reports a 17.0 percent diarisation error rate for community-1 on the AMI meeting set. pyannote also sells a more accurate model, but it runs on their servers, which defeats the point of self-hosting.

Accents and code switching: what to expect

Whisper was trained on 680,000 hours of audio, 65 percent of it English. Its model card says performance is uneven across languages and differs across accents and dialects, and that it can hallucinate text nobody said.

• Indian English. The Svarah benchmark (117 speakers from 65 locations in India) found Indian speakers poorly represented in standard English test sets and "clear scope for improvement on Indian accents" in the systems it tested.
• Arabic and English in one call. The Mixat study of Emirati speakers found pre-trained Arabic and multilingual models struggle with the dialect, and that switching to English mid sentence adds a further challenge.
• Hindi or Tamil mixed with English. Common on Indian sales and support calls, and the same caution applies.

So test before you trust it. Transcribe 30 to 50 of your own recordings, have a person correct them, and measure the word error rate by speaker group. If it is too high for your use, the options are a larger model, a model fine-tuned on your accent or language pair, or a person reviewing the transcripts that matter.

A setup that keeps recordings in the building

1. Copy recordings from the phone system to a server on your network.
2. Run faster-whisper with voice activity filtering, then diarisation if the calls are mono.
3. Store transcripts beside the audio, with the same access rules as the recordings.
4. Send summaries or CRM notes to a local LLM on the same server, as covered in our voice AI Labs track.
5. Delete audio on the schedule your retention policy sets.

Getting it running for your calls

We set up self hosted transcription as part of our private AI deployment work, and connect it to call flows through our voice AI agents service. Bring a sample of real calls and we will tell you what accuracy to expect before you buy hardware. Book a consultation.