Amiya Pani

Speech-to-Text Models: Pros, Cons, and How to Fine-Tune One for Your Domain (Like Movies)

2026-11-08T00:00:00+05:30

Speech-to-text (STT) has quietly become one of the most useful building blocks in AI. Meeting notes, call-centre analytics, subtitles, voice assistants, dubbing and translation pipelines all start with the same step: turn audio into accurate text.

The good news is that modern foundation models are remarkably good at this. The bad news is that "remarkably good" on a benchmark often turns into "frustratingly wrong" on your audio: the character names, the accents, the background music, the jargon.

In this post I'll cover:

The main types of STT models, with their pros and cons
Why general models struggle with domain-specific audio
Indian and regional languages: what works today and where the gaps are
A practical ladder of fixes, from zero-training tricks to full fine-tuning
A worked example: adapting a foundation model to movie dialogue, including Indian cinema

The Three Families of STT Models

Almost every STT option today falls into one of three buckets.

1. Managed cloud APIs

Examples: Google Cloud Speech-to-Text, AWS Transcribe, Azure AI Speech, Deepgram, AssemblyAI, and the transcription endpoints from OpenAI and ElevenLabs.

2. Open-weight foundation models (self-hosted)

Examples: OpenAI Whisper (and faster variants such as Whisper Turbo and faster-whisper), NVIDIA Parakeet and Canary (NeMo), and Meta's wav2vec 2.0 and MMS family.

3. On-device / edge models

Examples: whisper.cpp, small quantised Whisper or Parakeet models, Vosk, and the speech engines built into iOS and Android.

Pros and cons at a glance

	Cloud APIs	Open-weight (self-hosted)	On-device
Accuracy out of the box	Very high on general audio	High; the best models rival the cloud	Lower; smaller models trade accuracy for size
Domain customisation	Limited: custom vocabulary or phrase boosting at best	Full control: prompt, fine-tune, or retrain	Possible, but you train elsewhere and shrink the model
Cost at scale	Pay per minute; adds up fast	GPU cost only; cheap at high volume	Free per use once shipped
Data privacy	Audio leaves your network	Stays in your infrastructure	Never leaves the device
Latency / streaming	Good, real-time options available	Depends on your setup; Whisper is batch-first	Excellent, no network hop
Operational effort	Minimal	You own GPUs, scaling, and monitoring	You own app integration and model updates
Languages	Broad	Broad (Whisper covers ~100 languages)	Usually narrower

My rule of thumb:

Start with a cloud API when you need to prove value quickly and your audio is "normal" (clear speech, common vocabulary).
Move to an open-weight model when cost, privacy, or domain accuracy starts to hurt. This is the only family where you can truly teach the model your domain.
Go on-device when latency or privacy is non-negotiable, such as live captions or offline apps.

Known weak spots (whatever model you pick)

Hallucinations on silence and music. Generative models like Whisper sometimes "invent" text during long pauses or music, a well-known issue that matters a lot for movies.
Proper nouns. Character names, product names, and place names get spelled phonetically or replaced with common words.
Overlapping speech. Two people talking at once usually becomes one garbled sentence. Speaker diarisation (who spoke when) is a separate problem.
Accents and code-switching. Mixed-language speech (for example, Hindi–English "Hinglish") is still hard for most general models.
Noise and effects. Explosions, crowd noise, and background scores drown out dialogue.

Why a General Model Isn't Enough for Your Domain

Foundation models are trained on huge amounts of general audio. That gives them broad ability but no knowledge of:

Your vocabulary: the names of characters, franchises, actors, or internal products
Your acoustic conditions: film mixes, call-centre lines, factory floors
Your style rules: whether you want "gonna" or "going to", numbers as digits or words, and tags like [laughs]

Movies are a great example because they combine almost every weak spot at once: music, effects, whispers, shouting, accents, overlapping dialogue, and invented names.

Indian and Regional Languages: The Real Test

India has 22 scheduled languages, hundreds of dialects, and a habit of mixing them mid-sentence. For speech AI, that makes it one of the hardest and most interesting markets in the world. If your users are in India, an STT model that only shines in English is not enough.

What's available today

Option	Indian language support	Good for
Whisper (OpenAI, open-weight)	Hindi, Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Urdu and a few others, but accuracy drops sharply outside Hindi	A quick multilingual baseline and a strong starting point for fine-tuning
AI4Bharat models (IIT Madras, open-source)	Built specifically for Indian languages, e.g. IndicWhisper and IndicConformer, trained on large Indian speech datasets	Best open starting point for most Indic languages
Indian AI companies (e.g. Sarvam AI)	STT APIs focused on Indian languages and code-mixed speech, some with speech-to-English translation	Production use when you want a managed, India-focused service
Bhashini (Government of India)	National language platform offering speech, translation and transliteration services across Indian languages	Public-sector and citizen-facing apps
Global cloud APIs (Google, Azure, AWS)	Cover the major Indian languages; depth varies by language and provider	Teams already on that cloud; always test per language
Meta MMS (open-weight)	Covers 1,000+ languages, including many low-resource Indian ones	Languages nobody else supports, as a starting point

Language lists and accuracy change quickly in this space, so treat this as a map, not a scorecard. Always benchmark on your own audio, in each language you need.

What makes Indian languages hard

Code-mixing is the norm. "Kal meeting reschedule kar dena" is one sentence in two languages. Hinglish, Tanglish (Tamil–English), Benglish and others need models trained on mixed speech, not on clean monolingual text.
Script choice. Should Hinglish be written in Devanagari (कल मीटिंग), Roman script (kal meeting), or a mix? There is no single right answer, but your training data and output must follow one consistent rule, or accuracy numbers become meaningless.
Low-resource languages. Hindi has plenty of training data. Languages like Odia, Assamese, Konkani, Manipuri, or tribal languages have far less, and general models show it.
Dialects and accents. Bhojpuri-influenced Hindi, Kongu Tamil, and Vidarbha Marathi can all trip up a model trained on "standard" speech.
Inefficient tokenisation. Many multilingual models split Indic scripts into many more tokens than English. That makes transcription slower and more expensive, and it is one reason India-specific models often outperform global ones.
Telephony audio. A lot of Indian voice data comes from 8 kHz phone calls (support lines, IVRs), which sounds very different from the clean 16 kHz audio most models are trained on.

Open datasets worth knowing

AI4Bharat: IndicVoices, Kathbath, Shrutilipi
Google and IISc: Project Vaani, which collects speech from districts across India
Mozilla Common Voice and Google FLEURS: multilingual sets with several Indian languages

Check each dataset's licence before using it commercially.

The Customisation Ladder: Cheapest Fix First

Don't jump straight to fine-tuning. Climb this ladder and stop as soon as accuracy is good enough.

Step 1: Clean up the audio

Often the cheapest win. Run a voice activity detector (VAD) to skip silence and music (this alone cuts many hallucinations). If you have access to the stems, use the dialogue track; otherwise a source-separation model such as Demucs can isolate vocals.

Step 2: Prompt the model with context

Whisper accepts an initial_prompt. Feeding it the character names and a sample of the expected style nudges spelling without any training:

import whisper

model = whisper.load_model("large-v3")
result = model.transcribe(
    "scene_042.wav",
    initial_prompt="Characters: Kaveri, Rudransh, Inspector Malhotra. Setting: Mumbai docks.",
)
print(result["text"])

Cloud APIs offer a similar feature called custom vocabulary or phrase hints.

Step 3: Fix the text afterwards with an LLM

Pass the raw transcript, plus a glossary of names and terms, to an LLM and ask it to correct spellings without changing the meaning. This works well for proper nouns, but be careful: an over-eager LLM can "improve" dialogue that was supposed to be ungrammatical.

Step 4: Parameter-efficient fine-tuning (LoRA)

When steps 1–3 aren't enough, fine-tune the model on your own labelled audio. With LoRA (Low-Rank Adaptation) you freeze the original model and train small adapter layers, often under 1% of the parameters. Benefits:

Fits on a single modern GPU
Much lower risk of the model "forgetting" its general skills
Adapters are small files, so you can keep one per domain (movies, sports, medical) on top of the same base model

Step 5: Full fine-tuning

Update every weight in the model. It needs more data, more GPU, and more care, but it gives the biggest gains when your domain is very different from general speech (for example, a low-resource language or heavily accented audio).

Worked Example: Teaching Whisper to Understand Movies

Let's fine-tune Whisper with LoRA on movie dialogue.

1. Get legally usable data

This is the step people skip, and it's the one that matters most. Only use films and subtitles you have the rights to use for training: your own productions, licensed catalogues, or openly licensed content. Studio audio scraped from the internet is a legal risk, not a dataset.

2. Build audio–text pairs from subtitles

Subtitle files (.srt) already contain timestamps and text, which makes them a natural starting point:

Extract audio as 16 kHz mono, the format Whisper expects: ffmpeg -i movie.mkv -ac 1 -ar 16000 movie.wav
Cut clips using each subtitle's start and end time. Merge neighbouring lines so clips stay under Whisper's 30-second window.
Clean the text. Remove [MUSIC], ♪, speaker labels like JOHN:, HTML tags, and line breaks.
Fix timing drift. Subtitles are often a few hundred milliseconds off. A forced-alignment tool (for example, WhisperX) can re-align them.

Important: subtitles are not verbatim transcripts. They are shortened for reading speed and often drop filler words. For the best results, hand-correct a few hours of clips into true verbatim text, at least for your test set.

Organise the clips like this:

movie_clips/
  train/  metadata.csv   clip_0001.wav ...
  test/   metadata.csv   clip_9001.wav ...

where each metadata.csv has the columns file_name,transcription.

Split by movie, not by clip. If clips from the same film appear in both train and test, your accuracy numbers will look far better than they really are.

Doing this for Indian cinema

Indian films are a goldmine for multilingual speech data, with a few extra wrinkles:

Songs are a large share of runtime. Detect and exclude song sequences from dialogue training, or build a separate lyrics model. Music is the biggest source of hallucinations.
Dubbed versions give you parallel audio. Many films release in Hindi, Tamil, Telugu, Kannada and Malayalam with the same scenes and timing. With the rights, that's a ready-made multilingual dataset, aligned by scene.
Dialogue is often code-mixed. Decide your script convention for mixed lines before labelling, and write it down for every annotator.
Start from an Indic model. For Tamil or Malayalam, an AI4Bharat checkpoint usually beats vanilla Whisper as a base. The fine-tuning approach below works the same way; you change the base model and the language setting.

3. Fine-tune with Hugging Face and PEFT

from dataclasses import dataclass

from datasets import Audio, load_dataset
from peft import LoraConfig, get_peft_model
from transformers import (
    Seq2SeqTrainer,
    Seq2SeqTrainingArguments,
    WhisperForConditionalGeneration,
    WhisperProcessor,
)

BASE = "openai/whisper-small"   # start small; move up once the pipeline works
LANGUAGE = "english"            # e.g. "hindi", "tamil", "telugu" for Indian-language films

processor = WhisperProcessor.from_pretrained(BASE, language=LANGUAGE, task="transcribe")
model = WhisperForConditionalGeneration.from_pretrained(BASE)

# 1. Load and prepare the data
ds = load_dataset("audiofolder", data_dir="movie_clips")
ds = ds.cast_column("audio", Audio(sampling_rate=16_000))

def prepare(example):
    audio = example["audio"]
    example["input_features"] = processor.feature_extractor(
        audio["array"], sampling_rate=16_000
    ).input_features[0]
    example["labels"] = processor.tokenizer(example["transcription"]).input_ids
    return example

ds = ds.map(prepare, remove_columns=ds["train"].column_names)

# 2. Pad audio features and labels separately
@dataclass
class SpeechCollator:
    processor: WhisperProcessor

    def __call__(self, features):
        inputs = [{"input_features": f["input_features"]} for f in features]
        batch = self.processor.feature_extractor.pad(inputs, return_tensors="pt")

        labels = [{"input_ids": f["labels"]} for f in features]
        labels = self.processor.tokenizer.pad(labels, return_tensors="pt")
        ids = labels["input_ids"].masked_fill(labels.attention_mask.ne(1), -100)
        if (ids[:, 0] == model.config.decoder_start_token_id).all():
            ids = ids[:, 1:]
        batch["labels"] = ids
        return batch

# 3. Add LoRA adapters: only these small layers are trained
lora = LoraConfig(r=32, lora_alpha=64, target_modules=["q_proj", "v_proj"], lora_dropout=0.05)
model = get_peft_model(model, lora)
model.print_trainable_parameters()   # typically well under 1% of the model

# 4. Train
args = Seq2SeqTrainingArguments(
    output_dir="whisper-movies-lora",
    per_device_train_batch_size=16,
    learning_rate=1e-3,               # LoRA tolerates a higher LR than full fine-tuning
    warmup_steps=50,
    max_steps=2000,
    fp16=True,
    eval_strategy="steps",
    eval_steps=250,
    save_steps=250,
    logging_steps=25,
    remove_unused_columns=False,      # required with PEFT
    label_names=["labels"],
)

trainer = Seq2SeqTrainer(
    model=model,
    args=args,
    train_dataset=ds["train"],
    eval_dataset=ds["test"],
    data_collator=SpeechCollator(processor),
)
trainer.train()
model.save_pretrained("whisper-movies-lora")   # saves only the small adapter

4. Measure: word error rate (WER)

The standard metric is word error rate: the share of words the model got wrong (substituted, deleted, or inserted). Lower is better. Compare the base model and your fine-tuned model on the same held-out movies:

import evaluate

wer = evaluate.load("wer")
score = wer.compute(predictions=model_outputs, references=ground_truth)
print(f"WER: {score:.2%}")

Beyond the overall number, check the things you actually care about:

Name accuracy: are the character names now spelled correctly?
Hallucination rate: does the model still invent text during music-only scenes?
No regressions: run a slice of general audio too, to make sure the model hasn't forgotten everyday speech.
Per-language scores: for multilingual or code-mixed content, report WER separately for each language. One overall number can hide a model that is great at Hindi and poor at Malayalam. For languages like Tamil and Malayalam, where one long word can carry a whole phrase, a single wrong letter fails the entire word, so also look at character error rate (CER).

5. Deploy

Load the base model, attach the adapter, and optionally merge them into a single model for faster inference. Then serve it with an optimised runtime such as faster-whisper (CTranslate2) or behind a GPU inference server.

Beyond Transcription: Where This Goes Next

Once you have a domain-tuned STT model, it becomes the first stage of bigger pipelines:

Subtitles and captioning: transcription plus timing plus line-length rules
Translation and dubbing: STT → machine translation → text-to-speech
Search and analytics: making a whole film catalogue searchable by dialogue
Content moderation and compliance: flagging language for age ratings

Each stage benefits from the same lesson: general models give you a strong start, and your domain data is what makes them truly useful.

Frequently Asked Questions

What is the best speech-to-text model?

There is no single best model. Cloud APIs are the fastest way to get high accuracy on clear, general audio. Open-weight models such as Whisper are best when you need privacy, lower cost at scale, or the ability to fine-tune on your own data. On-device models are best for real-time or offline use. Always benchmark on your own audio.

Which speech-to-text model works best for Hindi and other Indian languages?

Models built for India, such as AI4Bharat's IndicWhisper and IndicConformer or India-focused services like Sarvam AI and Bhashini, usually outperform global models on Indian languages and code-mixed speech like Hinglish. Whisper works reasonably well for Hindi but drops off for languages with less training data.

Can I fine-tune Whisper on my own data?

Yes. Using LoRA (Low-Rank Adaptation), you train small adapter layers on top of a frozen Whisper model. It fits on a single modern GPU and keeps the model's general skills. A few hours of clean, accurately transcribed domain audio is a practical starting point.

Why does Whisper make up text during music or silence?

Whisper is a generative model, so it can produce plausible-sounding text even when no one is speaking. Running a voice activity detector (VAD) to skip non-speech segments, and isolating the dialogue track, removes most of these hallucinations.

How do you measure speech-to-text accuracy?

The standard metric is word error rate (WER): the share of words substituted, deleted or inserted compared with a correct transcript. For Indian languages with long compound words, such as Tamil and Malayalam, also track character error rate (CER).

Key Takeaways

Pick the model family for your constraints: cloud for speed to market, open-weight for control and cost at scale, on-device for latency and privacy.
For India, test language by language: India-focused models (AI4Bharat, Sarvam, Bhashini) often beat global ones on Indic and code-mixed speech. Pick a script convention early and stick to it.
Climb the customisation ladder: audio clean-up → prompting → LLM post-correction → LoRA → full fine-tune.
Data quality beats data quantity: clean, correctly licensed, verbatim transcripts matter more than raw hours.
Evaluate honestly: split by movie, track WER and the errors your users notice, and watch for regressions.

If you're working on speech AI, whether transcription, translation, or voice, I'd love to compare notes. Reach out on LinkedIn or by email.

How to Use Windows Apps Like Power BI and Notepad++ on a Mac

2026-05-18T18:00:00+05:30

Got a Mac but your work needs you to use Power BI or Notepad++? You're not alone. Many popular tools were originally built only for Windows, which leaves Mac users a bit stuck.

The good news - there are several easy ways around this. Let me walk you through each one, starting with the simplest.

Option 1: Just Use the Website (No Setup Needed)

Before doing anything else, check if the app has a website version.

Power BI has a free website version at app.powerbi.com. Just open it in your browser - Chrome, Safari, Firefox, any of them work. You can view reports, make edits, and share dashboards without installing anything on your Mac.

Think of it like using Google Docs instead of Microsoft Word - same idea, works in the browser.

Best for: People who mostly need to view or share Power BI reports rather than build them from scratch.

Option 2: Notepad++ - Switch to a Free Mac App Instead

Notepad++ is just a text editor - a tool for opening and editing files. On Mac, there are free apps that do the exact same thing (and honestly do it better).

The most popular one is Visual Studio Code (everyone just calls it VS Code). It's completely free, easy to use, and works just like Notepad++ - you open a file, edit it, save it. That's it.

To install VS Code: 1. Go to code.visualstudio.com 2. Click the big download button 3. Open the downloaded file and drag it into your Applications folder 4. Done

Another option is Sublime Text - also free to try, very fast and simple, great if you just want something lightweight.

Best for: Anyone using Notepad++ to read, write, or edit text files. VS Code is a direct swap - no learning curve.

Option 3: CrossOver - Run Windows Apps on Your Mac Directly

CrossOver is an app for Mac that lets you install and run many Windows applications without needing Windows itself. It basically tricks your Mac into thinking it's running Windows, just enough for the app to work.

How to use it: 1. Go to codeweavers.com/crossover and download it (there's a free 14-day trial) 2. Open CrossOver and search for the app you want (e.g. "Notepad++") 3. Click Install 4. The app opens on your Mac just like any other app

It works well for simpler apps like Notepad++. For heavier apps like Power BI Desktop, it can be a bit unreliable - sometimes it works, sometimes it doesn't.

Cost: Free trial for 14 days, then around ₹6,000/year (or ~$74/year)

Best for: Occasional use of lighter Windows apps. Not ideal for Power BI.

Option 4: Parallels Desktop - The Best Way to Run Power BI on Mac

This is the most reliable solution, especially for Power BI Desktop.

Parallels Desktop creates a "Windows room" inside your Mac. Imagine your Mac as a house - Parallels adds a separate room inside it that runs Windows. You can switch between your normal Mac and this Windows room anytime.

Once Windows is running inside Parallels, you install Power BI Desktop (or any Windows app) exactly as you normally would on a Windows computer. It works perfectly because it literally is Windows.

The cool part - Coherence Mode: There's a setting called Coherence Mode where Power BI Desktop appears as a regular window on your Mac screen - no "Windows room" visible at all. It just sits alongside your other Mac apps like it belongs there.

How to get started: 1. Go to parallels.com and download Parallels Desktop (free 14-day trial) 2. It will guide you to download Windows automatically - just follow the steps 3. Once Windows is set up, open it and install Power BI Desktop from Microsoft's website 4. Turn on Coherence Mode and use Power BI just like any Mac app

Cost: Around ₹7,500/year (or ~$99/year). A one-time purchase option is also available.

Best for: Anyone who regularly needs Power BI Desktop or other heavy Windows-only apps.

Option 5: VMware Fusion - Same as Parallels but Free

VMware Fusion does the same thing as Parallels - it creates a Windows environment inside your Mac. The great news is that it's completely free for personal use as of 2024.

It's slightly more steps to set up than Parallels, but once it's running, it works just as well.

Go to vmware.com/products/fusion and download Fusion
Create a new Windows setup inside it
Install your Windows apps as normal

Cost: Free for personal use

Best for: Anyone who wants the full Windows experience on their Mac without paying for Parallels.

Option 6: Connect to a Windows Computer Remotely

If you already have a Windows computer at home, or your workplace has Windows machines, you can control that computer from your Mac - even from a different room or a different city.

Microsoft has a free app for this called Microsoft Remote Desktop. You install it on your Mac, type in the address of your Windows computer, and your Mac screen shows the Windows desktop. You can open Power BI, use Notepad++, do anything - all remotely.

How to set it up: 1. Search "Microsoft Remote Desktop" in the Mac App Store and install it (free) 2. Ask your IT team for your Windows computer's address, or set it up at home 3. Connect - your Windows desktop appears on your Mac screen

Best for: Office workers whose company has Windows computers, or people who have a Windows PC at home.

So Which One Should You Use?

What you need	Best option
Just view Power BI reports	Power BI website (free)
Build Power BI reports regularly	Parallels or VMware Fusion
Replace Notepad++	VS Code (free, works great)
Run a Windows app occasionally	CrossOver
Already have a Windows PC	Microsoft Remote Desktop (free)
Want everything Windows for free	VMware Fusion

Quick Summary

For Power BI - use the website for viewing, or Parallels/VMware if you need the full desktop app
For Notepad++ - just install VS Code, it does the same job and it's free
For anything else Windows - CrossOver for simple apps, Parallels/VMware for everything else

The Mac is a brilliant computer for almost everything - and with these options, Windows-only apps don't need to slow you down anymore.

Have a specific app you're trying to get working on your Mac? Feel free to reach out - happy to help.

Welcome to My Blog - Thoughts on Data, AI & Automation

2026-05-18T09:00:00+05:30

Welcome to my corner of the internet!

I'm Amiya Pani - a Data Engineer and AI Specialist based in Mumbai, India, with over 8 years of experience building data pipelines, intelligent automation systems, and AI-powered applications.

Why a Blog?

I've spent years solving complex data engineering challenges, designing automation frameworks, and deploying machine learning models in production. Along the way I've accumulated a lot of lessons - things I wish I'd known earlier, patterns that work, pitfalls to avoid.

This blog is where I'll share that knowledge.

What to Expect

Here's what I plan to cover:

Data Engineering - ETL pipeline design, Databricks, Snowflake, AWS Glue, Apache Airflow tips and patterns
AI & GenAI - Practical applications of LLMs, RAG pipelines, prompt engineering, OpenAI API integration
Python Automation - Scripts, patterns, and real-world automation use cases
RPA & Intelligent Automation - UiPath, Power Automate, Alteryx workflows
Career & Tools - Tools I use, lessons from enterprise projects, and thoughts on the data landscape

A Bit About Me

Currently a Lead Analyst at Crisil Limited, working with clients like Allianz Global Investors and Citi Bank on performance reporting automation, ETL pipelines, and risk analytics.

Previously at JP Morgan, Capgemini, ACL Digital, and Healthmeter - building everything from HR chatbots and IVR systems to regulatory reporting automation and inventory management with computer vision.

I hold certifications in AWS, Azure AI, Snowflake, Neo4j, UiPath, and more - and I'm currently pursuing an MBA in Business Analytics.

Stay tuned - first technical post coming soon. In the meantime, feel free to reach out on LinkedIn or drop me an email.