Speech-to-Text Models: Pros, Cons, and How to Fine-Tune One for Your Domain (Like Movies)

Speech-to-text (STT) has quietly become one of the most useful building blocks in AI. Meeting notes, call-centre analytics, subtitles, voice assistants, dubbing and translation pipelines all start with the same step: turn audio into accurate text.

The good news is that modern foundation models are remarkably good at this. The bad news is that "remarkably good" on a benchmark often turns into "frustratingly wrong" on your audio: the character names, the accents, the background music, the jargon.

In this post I'll cover:

1. The main types of STT models, with their pros and cons
2. Why general models struggle with domain-specific audio
3. Indian and regional languages: what works today and where the gaps are
4. A practical ladder of fixes, from zero-training tricks to full fine-tuning
5. A worked example: adapting a foundation model to movie dialogue, including Indian cinema

The Three Families of STT Models

Almost every STT option today falls into one of three buckets.

1. Managed cloud APIs

Examples: Google Cloud Speech-to-Text, AWS Transcribe, Azure AI Speech, Deepgram, AssemblyAI, and the transcription endpoints from OpenAI and ElevenLabs.

2. Open-weight foundation models (self-hosted)

Examples: OpenAI Whisper (and faster variants such as Whisper Turbo and faster-whisper), NVIDIA Parakeet and Canary (NeMo), and Meta's wav2vec 2.0 and MMS family.

3. On-device / edge models

Examples: whisper.cpp, small quantised Whisper or Parakeet models, Vosk, and the speech engines built into iOS and Android.

Pros and cons at a glance

My rule of thumb:

• Start with a cloud API when you need to prove value quickly and your audio is "normal" (clear speech, common vocabulary).
• Move to an open-weight model when cost, privacy, or domain accuracy starts to hurt. This is the only family where you can truly teach the model your domain.
• Go on-device when latency or privacy is non-negotiable, such as live captions or offline apps.

Known weak spots (whatever model you pick)

• Hallucinations on silence and music. Generative models like Whisper sometimes "invent" text during long pauses or music, a well-known issue that matters a lot for movies.
• Proper nouns. Character names, product names, and place names get spelled phonetically or replaced with common words.
• Overlapping speech. Two people talking at once usually becomes one garbled sentence. Speaker diarisation (who spoke when) is a separate problem.
• Accents and code-switching. Mixed-language speech (for example, Hindi–English "Hinglish") is still hard for most general models.
• Noise and effects. Explosions, crowd noise, and background scores drown out dialogue.

Why a General Model Isn't Enough for Your Domain

Foundation models are trained on huge amounts of general audio. That gives them broad ability but no knowledge of:

• Your vocabulary: the names of characters, franchises, actors, or internal products
• Your acoustic conditions: film mixes, call-centre lines, factory floors
• Your style rules: whether you want "gonna" or "going to", numbers as digits or words, and tags like [laughs]

Movies are a great example because they combine almost every weak spot at once: music, effects, whispers, shouting, accents, overlapping dialogue, and invented names.

Indian and Regional Languages: The Real Test

India has 22 scheduled languages, hundreds of dialects, and a habit of mixing them mid-sentence. For speech AI, that makes it one of the hardest and most interesting markets in the world. If your users are in India, an STT model that only shines in English is not enough.

What's available today

Language lists and accuracy change quickly in this space, so treat this as a map, not a scorecard. Always benchmark on your own audio, in each language you need.

What makes Indian languages hard

• Code-mixing is the norm. "Kal meeting reschedule kar dena" is one sentence in two languages. Hinglish, Tanglish (Tamil–English), Benglish and others need models trained on mixed speech, not on clean monolingual text.
• Script choice. Should Hinglish be written in Devanagari (कल मीटिंग), Roman script (kal meeting), or a mix? There is no single right answer, but your training data and output must follow one consistent rule, or accuracy numbers become meaningless.
• Low-resource languages. Hindi has plenty of training data. Languages like Odia, Assamese, Konkani, Manipuri, or tribal languages have far less, and general models show it.
• Dialects and accents. Bhojpuri-influenced Hindi, Kongu Tamil, and Vidarbha Marathi can all trip up a model trained on "standard" speech.
• Inefficient tokenisation. Many multilingual models split Indic scripts into many more tokens than English. That makes transcription slower and more expensive, and it is one reason India-specific models often outperform global ones.
• Telephony audio. A lot of Indian voice data comes from 8 kHz phone calls (support lines, IVRs), which sounds very different from the clean 16 kHz audio most models are trained on.

Open datasets worth knowing

• AI4Bharat: IndicVoices, Kathbath, Shrutilipi
• Google and IISc: Project Vaani, which collects speech from districts across India
• Mozilla Common Voice and Google FLEURS: multilingual sets with several Indian languages

Check each dataset's licence before using it commercially.

The Customisation Ladder: Cheapest Fix First

Don't jump straight to fine-tuning. Climb this ladder and stop as soon as accuracy is good enough.

Step 1: Clean up the audio

Often the cheapest win. Run a voice activity detector (VAD) to skip silence and music (this alone cuts many hallucinations). If you have access to the stems, use the dialogue track; otherwise a source-separation model such as Demucs can isolate vocals.

Step 2: Prompt the model with context

Whisper accepts an initial_prompt. Feeding it the character names and a sample of the expected style nudges spelling without any training:

import whisper

model = whisper.load_model("large-v3")
result = model.transcribe(
    "scene_042.wav",
    initial_prompt="Characters: Kaveri, Rudransh, Inspector Malhotra. Setting: Mumbai docks.",
)
print(result["text"])

Cloud APIs offer a similar feature called custom vocabulary or phrase hints.

Step 3: Fix the text afterwards with an LLM

Pass the raw transcript, plus a glossary of names and terms, to an LLM and ask it to correct spellings without changing the meaning. This works well for proper nouns, but be careful: an over-eager LLM can "improve" dialogue that was supposed to be ungrammatical.

Step 4: Parameter-efficient fine-tuning (LoRA)

When steps 1–3 aren't enough, fine-tune the model on your own labelled audio. With LoRA (Low-Rank Adaptation) you freeze the original model and train small adapter layers, often under 1% of the parameters. Benefits:

• Fits on a single modern GPU
• Much lower risk of the model "forgetting" its general skills
• Adapters are small files, so you can keep one per domain (movies, sports, medical) on top of the same base model

Step 5: Full fine-tuning

Update every weight in the model. It needs more data, more GPU, and more care, but it gives the biggest gains when your domain is very different from general speech (for example, a low-resource language or heavily accented audio).

Worked Example: Teaching Whisper to Understand Movies

Let's fine-tune Whisper with LoRA on movie dialogue.

1. Get legally usable data

This is the step people skip, and it's the one that matters most. Only use films and subtitles you have the rights to use for training: your own productions, licensed catalogues, or openly licensed content. Studio audio scraped from the internet is a legal risk, not a dataset.

2. Build audio–text pairs from subtitles

Subtitle files (.srt) already contain timestamps and text, which makes them a natural starting point:

1. Extract audio as 16 kHz mono, the format Whisper expects: ffmpeg -i movie.mkv -ac 1 -ar 16000 movie.wav
2. Cut clips using each subtitle's start and end time. Merge neighbouring lines so clips stay under Whisper's 30-second window.
3. Clean the text. Remove [MUSIC], ♪, speaker labels like JOHN:, HTML tags, and line breaks.
4. Fix timing drift. Subtitles are often a few hundred milliseconds off. A forced-alignment tool (for example, WhisperX) can re-align them.

Important: subtitles are not verbatim transcripts. They are shortened for reading speed and often drop filler words. For the best results, hand-correct a few hours of clips into true verbatim text, at least for your test set.

Organise the clips like this:

movie_clips/
  train/  metadata.csv   clip_0001.wav ...
  test/   metadata.csv   clip_9001.wav ...

where each metadata.csv has the columns file_name,transcription.

Split by movie, not by clip. If clips from the same film appear in both train and test, your accuracy numbers will look far better than they really are.

Doing this for Indian cinema

Indian films are a goldmine for multilingual speech data, with a few extra wrinkles:

• Songs are a large share of runtime. Detect and exclude song sequences from dialogue training, or build a separate lyrics model. Music is the biggest source of hallucinations.
• Dubbed versions give you parallel audio. Many films release in Hindi, Tamil, Telugu, Kannada and Malayalam with the same scenes and timing. With the rights, that's a ready-made multilingual dataset, aligned by scene.
• Dialogue is often code-mixed. Decide your script convention for mixed lines before labelling, and write it down for every annotator.
• Start from an Indic model. For Tamil or Malayalam, an AI4Bharat checkpoint usually beats vanilla Whisper as a base. The fine-tuning approach below works the same way; you change the base model and the language setting.

3. Fine-tune with Hugging Face and PEFT

from dataclasses import dataclass

from datasets import Audio, load_dataset
from peft import LoraConfig, get_peft_model
from transformers import (
    Seq2SeqTrainer,
    Seq2SeqTrainingArguments,
    WhisperForConditionalGeneration,
    WhisperProcessor,
)

BASE = "openai/whisper-small"   # start small; move up once the pipeline works
LANGUAGE = "english"            # e.g. "hindi", "tamil", "telugu" for Indian-language films

processor = WhisperProcessor.from_pretrained(BASE, language=LANGUAGE, task="transcribe")
model = WhisperForConditionalGeneration.from_pretrained(BASE)

# 1. Load and prepare the data
ds = load_dataset("audiofolder", data_dir="movie_clips")
ds = ds.cast_column("audio", Audio(sampling_rate=16_000))

def prepare(example):
    audio = example["audio"]
    example["input_features"] = processor.feature_extractor(
        audio["array"], sampling_rate=16_000
    ).input_features[0]
    example["labels"] = processor.tokenizer(example["transcription"]).input_ids
    return example

ds = ds.map(prepare, remove_columns=ds["train"].column_names)

# 2. Pad audio features and labels separately
@dataclass
class SpeechCollator:
    processor: WhisperProcessor

    def __call__(self, features):
        inputs = [{"input_features": f["input_features"]} for f in features]
        batch = self.processor.feature_extractor.pad(inputs, return_tensors="pt")

        labels = [{"input_ids": f["labels"]} for f in features]
        labels = self.processor.tokenizer.pad(labels, return_tensors="pt")
        ids = labels["input_ids"].masked_fill(labels.attention_mask.ne(1), -100)
        if (ids[:, 0] == model.config.decoder_start_token_id).all():
            ids = ids[:, 1:]
        batch["labels"] = ids
        return batch

# 3. Add LoRA adapters: only these small layers are trained
lora = LoraConfig(r=32, lora_alpha=64, target_modules=["q_proj", "v_proj"], lora_dropout=0.05)
model = get_peft_model(model, lora)
model.print_trainable_parameters()   # typically well under 1% of the model

# 4. Train
args = Seq2SeqTrainingArguments(
    output_dir="whisper-movies-lora",
    per_device_train_batch_size=16,
    learning_rate=1e-3,               # LoRA tolerates a higher LR than full fine-tuning
    warmup_steps=50,
    max_steps=2000,
    fp16=True,
    eval_strategy="steps",
    eval_steps=250,
    save_steps=250,
    logging_steps=25,
    remove_unused_columns=False,      # required with PEFT
    label_names=["labels"],
)

trainer = Seq2SeqTrainer(
    model=model,
    args=args,
    train_dataset=ds["train"],
    eval_dataset=ds["test"],
    data_collator=SpeechCollator(processor),
)
trainer.train()
model.save_pretrained("whisper-movies-lora")   # saves only the small adapter

4. Measure: word error rate (WER)

The standard metric is word error rate: the share of words the model got wrong (substituted, deleted, or inserted). Lower is better. Compare the base model and your fine-tuned model on the same held-out movies:

import evaluate

wer = evaluate.load("wer")
score = wer.compute(predictions=model_outputs, references=ground_truth)
print(f"WER: {score:.2%}")

Beyond the overall number, check the things you actually care about:

• Name accuracy: are the character names now spelled correctly?
• Hallucination rate: does the model still invent text during music-only scenes?
• No regressions: run a slice of general audio too, to make sure the model hasn't forgotten everyday speech.
• Per-language scores: for multilingual or code-mixed content, report WER separately for each language. One overall number can hide a model that is great at Hindi and poor at Malayalam. For languages like Tamil and Malayalam, where one long word can carry a whole phrase, a single wrong letter fails the entire word, so also look at character error rate (CER).

5. Deploy

Load the base model, attach the adapter, and optionally merge them into a single model for faster inference. Then serve it with an optimised runtime such as faster-whisper (CTranslate2) or behind a GPU inference server.

Beyond Transcription: Where This Goes Next

Once you have a domain-tuned STT model, it becomes the first stage of bigger pipelines:

• Subtitles and captioning: transcription plus timing plus line-length rules
• Translation and dubbing: STT → machine translation → text-to-speech
• Search and analytics: making a whole film catalogue searchable by dialogue
• Content moderation and compliance: flagging language for age ratings

Each stage benefits from the same lesson: general models give you a strong start, and your domain data is what makes them truly useful.

Frequently Asked Questions

What is the best speech-to-text model?

There is no single best model. Cloud APIs are the fastest way to get high accuracy on clear, general audio. Open-weight models such as Whisper are best when you need privacy, lower cost at scale, or the ability to fine-tune on your own data. On-device models are best for real-time or offline use. Always benchmark on your own audio.

Which speech-to-text model works best for Hindi and other Indian languages?

Models built for India, such as AI4Bharat's IndicWhisper and IndicConformer or India-focused services like Sarvam AI and Bhashini, usually outperform global models on Indian languages and code-mixed speech like Hinglish. Whisper works reasonably well for Hindi but drops off for languages with less training data.

Can I fine-tune Whisper on my own data?

Yes. Using LoRA (Low-Rank Adaptation), you train small adapter layers on top of a frozen Whisper model. It fits on a single modern GPU and keeps the model's general skills. A few hours of clean, accurately transcribed domain audio is a practical starting point.

Why does Whisper make up text during music or silence?

Whisper is a generative model, so it can produce plausible-sounding text even when no one is speaking. Running a voice activity detector (VAD) to skip non-speech segments, and isolating the dialogue track, removes most of these hallucinations.

How do you measure speech-to-text accuracy?

The standard metric is word error rate (WER): the share of words substituted, deleted or inserted compared with a correct transcript. For Indian languages with long compound words, such as Tamil and Malayalam, also track character error rate (CER).

Key Takeaways

• Pick the model family for your constraints: cloud for speed to market, open-weight for control and cost at scale, on-device for latency and privacy.
• For India, test language by language: India-focused models (AI4Bharat, Sarvam, Bhashini) often beat global ones on Indic and code-mixed speech. Pick a script convention early and stick to it.
• Climb the customisation ladder: audio clean-up → prompting → LLM post-correction → LoRA → full fine-tune.
• Data quality beats data quantity: clean, correctly licensed, verbatim transcripts matter more than raw hours.
• Evaluate honestly: split by movie, track WER and the errors your users notice, and watch for regressions.

If you're working on speech AI, whether transcription, translation, or voice, I'd love to compare notes. Reach out on LinkedIn or by email.