How to Run Whisper Speech-to-Text in the Browser

Whisper can run inside a browser through Transformers.js and ONNX WebAssembly. The audio stays in the tab during inference, while the model is downloaded and cached as an application dependency. That architecture offers a useful privacy boundary, but it is not the same as zero network traffic or server-grade streaming transcription.

A practical wrapper needs one cached pipeline, visible model progress, language control, long-audio chunking, object-URL cleanup, microphone permission handling, and a clear explanation of what “live” means when the browser periodically retranscribes accumulated recording data.

Separate model delivery from audio processing

The model, tokenizer, configuration, and ONNX runtime assets must reach the browser. In the current design, model files are allowed from the remote model repository and stored in the browser cache, while ONNX WASM files are served from the site. The user audio is represented by a local object URL and passed to the pipeline in the page.

On first use, users pay the model download and initialization cost. Later use may reuse cached model files, but private browsing, cleared site data, cache eviction, or a new model revision can trigger another download.

Create and cache one ASR pipeline

Pipeline construction is expensive, so cache the Promise rather than rebuilding it for every file. If initialization fails, reset that Promise to permit a real retry. A progress callback can distinguish model download, model readiness, inference, and completion.

The example deliberately uses one WASM thread for predictable compatibility. More threads can require cross-origin isolation and must be benchmarked across target browsers rather than enabled blindly.

const transformers = await import('@xenova/transformers/dist/transformers.min.js');

transformers.env.allowLocalModels = false;
transformers.env.useBrowserCache = true;
transformers.env.backends.onnx.wasm.wasmPaths = '/models/transformers/';
transformers.env.backends.onnx.wasm.numThreads = 1;

const transcriber = await transformers.pipeline(
  'automatic-speech-recognition',
  'Xenova/whisper-tiny',
  { progress_callback: reportModelProgress },
);

const result = await transcriber(URL.createObjectURL(file), {
  chunk_length_s: 30,
  stride_length_s: 5,
  language: selectedLanguage === 'auto' ? undefined : selectedLanguage,
});

Use overlapping chunks for long recordings

Whisper models operate on bounded audio windows. A 30-second chunk with a 5-second stride gives neighboring windows overlap, reducing the chance that a word at a hard boundary is lost. The pipeline reconciles the overlapping context into final text.

Long files still increase total compute time and memory pressure. Validate input type and size before model loading, show that transcription continues after the model reaches 100 percent, and avoid promising real-time speed on low-power devices.

• Auto language leaves language selection to the model wrapper.
• Explicit zh or en can reduce ambiguity when the recording language is known.
• Revoke the input object URL in finally, including error paths.
• Keep the pipeline cached, but release per-file URLs and UI previews.

Microphone transcription is periodic snapshot inference

MediaRecorder can emit one encoded chunk per second. At an interval, the application joins the chunks collected so far into a new File and runs the same file transcription function. The latest complete transcript replaces the previous preview.

This is easier to implement than a stateful streaming decoder, but compute grows as the accumulated recording grows because earlier audio is processed again. Prevent overlapping snapshot jobs, wait for an active job before finalizing, and always stop every MediaStream track.

const recorder = new MediaRecorder(stream, { mimeType });
const chunks: Blob[] = [];

recorder.ondataavailable = event => {
  if (event.data.size > 0) chunks.push(event.data);
};

recorder.start(1000);

const refreshId = window.setInterval(async () => {
  const snapshot = new File([new Blob(chunks, { type: mimeType })], 'live.webm');
  const result = await transcribeAudioFile(snapshot, { language: 'auto' });
  if (result.ok) setTranscript(result.text);
}, REFRESH_INTERVAL_MS);

Design microphone lifecycle and permissions carefully

Check MediaRecorder, getUserMedia, and a supported MIME type before showing the feature as available. Request echo cancellation and noise suppression when appropriate, but describe them as browser constraints rather than guarantees.

On stop, request the last data chunk, clear the refresh interval, stop the recorder, stop all tracks, run one final snapshot, and return the UI to idle. On denial or failure, perform the same cleanup before displaying a localized error.

Know where a tiny model is and is not enough

A tiny Whisper model prioritizes download size and browser feasibility. Accuracy can fall with noise, accents, multiple speakers, technical vocabulary, music, or distant microphones. Browser inference also varies by CPU, memory, and WASM support.

Use a larger model or controlled service when accuracy, diarization, timestamps, guaranteed latency, or centralized auditing is required. The UI should let users edit and copy the transcript rather than presenting model output as authoritative.

Key takeaways

Browser Whisper is a model-delivery and lifecycle problem as much as an inference call. Cache one pipeline, distinguish model traffic from audio handling, use overlapping chunks, serialize microphone snapshots, and clean up permissions and object URLs. Those details determine whether transcription feels trustworthy and usable.