When Vanilla Whisper Isn’t Enough
Standard OpenAI Whisper does a solid job transcribing audio. But when you need to know exactly when each word was spoken — or identify who said what in a multi-speaker recording — the base model falls short. Timestamps are sentence-level at best. Speaker separation doesn’t exist at all.
I hit this limit while processing interview recordings for a client’s internal knowledge base. They needed searchable transcripts with speaker attribution: something that could point to the exact timestamp of a specific statement, not just the paragraph it lived in. That’s when I started seriously comparing the available options.
Approach Comparison: What Are Your Options?
Vanilla Whisper (OpenAI)
The original model. Transcription quality is strong across many languages, but timestamps are segment-level — roughly sentence boundaries. No speaker diarization at all. For single-speaker video subtitles, it’s perfectly fine. For anything more complex, you’ll hit walls fast.
WhisperX
WhisperX is a community project built on top of Whisper that adds two critical capabilities: forced alignment for word-level timestamps using wav2vec2 alignment models, and speaker diarization via pyannote.audio. Everything runs locally — no audio leaves your machine.
AssemblyAI / Rev.ai / Deepgram (Cloud APIs)
Cloud speech APIs handle diarization and word timestamps well. The tradeoffs are real, though: you’re sending audio to a third party, paying per minute, and accepting their data retention policies. For a 10-hour archive of internal meetings, the privacy concern alone rules these out before you even look at the invoice.
Faster-Whisper
A faster reimplementation of Whisper using CTranslate2 — excellent throughput, and WhisperX actually uses it under the hood for the transcription step. Standalone faster-whisper doesn’t include the alignment or diarization pipeline, though.
Pros and Cons
WhisperX — Pros
- Word-level timestamps with high accuracy via forced alignment
- Speaker diarization with configurable min/max speaker counts
- Fully offline — audio never touches an external server
- GPU acceleration supported (CUDA), CPU fallback available
- Transcription covers 50+ languages
- Batch processing for faster throughput on long files
WhisperX — Cons
- Requires a Hugging Face account and token to download pyannote models (one-time setup, still local)
- Higher VRAM usage than vanilla Whisper when diarization is active
- Alignment models aren’t available for every language (falls back gracefully)
- Setup is more involved than a simple
pip install whisper - Speaker labels are generic (SPEAKER_00, SPEAKER_01) — no name resolution out of the box
Once you need speaker attribution or precise word timing in production, raw transcription is just the starting point. The pipeline described here covers what most real-world audio processing jobs actually require.
Recommended Setup
Run WhisperX in a Python virtual environment on a machine with at least 8GB RAM. GPU matters more than you might expect: on a 1-hour recording, large-v2 finishes in roughly 10–15 minutes on a mid-range CUDA GPU (RTX 3070 or better). The same job on CPU takes 60–90 minutes. For interactive use, GPU is worth the trouble. For overnight batch processing, CPU works fine.
Minimum requirements:
- Python 3.9+
- ffmpeg installed and in PATH
- Hugging Face account (free) with access to
pyannote/speaker-diarization-3.1 - 8GB+ RAM (16GB recommended for large models)
- Optional: NVIDIA GPU with CUDA 11.8+
Implementation Guide
Step 1: Install Dependencies
# Create and activate a virtual environment
python -m venv whisperx-env
source whisperx-env/bin/activate # Linux/macOS
# whisperx-env\Scripts\activate # Windows
# Install WhisperX
pip install whisperx
# Install ffmpeg (Ubuntu/Debian)
sudo apt install ffmpeg
# macOS
brew install ffmpeg
On a CUDA machine, verify your PyTorch installation matches your CUDA version before proceeding:
# For CUDA 11.8
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
Step 2: Get a Hugging Face Token
WhisperX uses pyannote.audio for diarization, and the pyannote models require accepting a user agreement on Hugging Face. Go to huggingface.co/pyannote/speaker-diarization-3.1, click “Agree and access repository”, then generate a token at huggingface.co/settings/tokens.
Step 3: Transcribe with Word-Level Timestamps
import whisperx
import json
device = "cuda" # or "cpu"
audio_file = "meeting.mp3"
batch_size = 16 # reduce if low on VRAM
compute_type = "float16" # use "int8" for CPU or low VRAM
# Step 1: Transcribe
model = whisperx.load_model("large-v2", device, compute_type=compute_type)
audio = whisperx.load_audio(audio_file)
result = model.transcribe(audio, batch_size=batch_size)
print(result["segments"]) # Segments without word-level timestamps yet
Step 4: Force Alignment for Word-Level Timestamps
# Step 2: Align whisper output
model_a, metadata = whisperx.load_align_model(
language_code=result["language"],
device=device
)
result = whisperx.align(
result["segments"],
model_a,
metadata,
audio,
device,
return_char_alignments=False
)
# Each word now has start/end timestamps
for segment in result["segments"]:
for word in segment["words"]:
print(f"{word['start']:.2f}s - {word['end']:.2f}s : {word['word']}")
The output looks like this:
0.42s - 0.68s : The
0.68s - 1.02s : deployment
1.02s - 1.34s : failed
1.34s - 1.56s : at
1.56s - 1.82s : midnight
Step 5: Speaker Diarization
# Step 3: Assign speaker labels
HF_TOKEN = "hf_your_token_here"
diarize_model = whisperx.DiarizationPipeline(
use_auth_token=HF_TOKEN,
device=device
)
# Optional: provide speaker count if known
diarize_segments = diarize_model(
audio,
min_speakers=2,
max_speakers=4
)
result = whisperx.assign_word_speakers(diarize_segments, result)
# Print results with speaker labels
for segment in result["segments"]:
speaker = segment.get("speaker", "UNKNOWN")
print(f"[{speaker}] {segment['start']:.1f}s: {segment['text']}")
Output example:
[SPEAKER_00] 0.4s: The deployment failed at midnight.
[SPEAKER_01] 3.2s: Did anyone check the rollback logs?
[SPEAKER_00] 7.8s: Yes, the database migration caused a timeout.
Step 6: Export to JSON or SRT
# Save full result as JSON
with open("transcript.json", "w", encoding="utf-8") as f:
json.dump(result["segments"], f, ensure_ascii=False, indent=2)
# Generate a simple SRT with speaker labels
def segments_to_srt(segments, output_path):
with open(output_path, "w", encoding="utf-8") as f:
for i, seg in enumerate(segments, start=1):
start = format_timestamp(seg["start"])
end = format_timestamp(seg["end"])
speaker = seg.get("speaker", "")
text = f"[{speaker}] {seg['text'].strip()}" if speaker else seg["text"].strip()
f.write(f"{i}\n{start} --> {end}\n{text}\n\n")
def format_timestamp(seconds):
h = int(seconds // 3600)
m = int((seconds % 3600) // 60)
s = int(seconds % 60)
ms = int((seconds % 1) * 1000)
return f"{h:02d}:{m:02d}:{s:02d},{ms:03d}"
segments_to_srt(result["segments"], "transcript.srt")
Putting It All Together as a CLI Script
import whisperx, json, sys
def transcribe_full(audio_path, hf_token, device="cpu"):
compute_type = "float16" if device == "cuda" else "int8"
model = whisperx.load_model("large-v2", device, compute_type=compute_type)
audio = whisperx.load_audio(audio_path)
result = model.transcribe(audio, batch_size=8)
model_a, metadata = whisperx.load_align_model(result["language"], device)
result = whisperx.align(result["segments"], model_a, metadata, audio, device)
diarize = whisperx.DiarizationPipeline(use_auth_token=hf_token, device=device)
diarize_segs = diarize(audio)
result = whisperx.assign_word_speakers(diarize_segs, result)
return result["segments"]
if __name__ == "__main__":
segments = transcribe_full(
audio_path=sys.argv[1],
hf_token=sys.argv[2],
device="cuda"
)
print(json.dumps(segments, ensure_ascii=False, indent=2))
Run it with:
python transcribe.py meeting.mp3 hf_your_token_here
A Few Things Worth Knowing Before You Ship
Diarization accuracy drops noticeably when speakers overlap or when audio quality is poor. Clean recordings with clear turn-taking work well. Noisy environments are trickier — run a noise reduction step first with something like noisereduce before feeding audio to WhisperX.
The alignment step adds processing time, but it’s worth it. Sentence-level timestamps tell you “somewhere in this paragraph.” Word-level timestamps tell you “exactly 4.2 seconds in.” For search indexing or transcript navigation, that precision changes everything.
