How to Build Browser OCR with ONNX Runtime Web

An OCR feature is not one neural network call. A useful browser pipeline must find text regions, correct upside-down crops, recognize variable-width lines, decode character probabilities, restore reading order, and keep all heavy work away from the UI thread.

This implementation uses three ONNX sessions—detection, orientation classification, and recognition—inside a module Web Worker. Images move into the worker as transferable bytes, preprocessing uses OffscreenCanvas, model sessions are cached, and progress is reported by request ID.

Split the pipeline into three model stages

The detection model produces a probability map for text pixels. Post-processing thresholds and dilates that map, finds connected components, filters weak regions, expands boxes, merges fragments on the same line, and sorts them into reading order.

Each detected crop then passes through a 0/180-degree classifier before the recognition model. Recognition returns per-step character scores, which are decoded with CTC-style blank and repeated-class removal. The final merge preserves line breaks and inserts spaces appropriately for English versus CJK text.

Transfer image bytes to a dedicated worker

The page maintains one worker and gives every request a unique ID. ArrayBuffer is included in the transfer list, moving ownership instead of cloning a potentially large image. Progress and result messages are filtered by ID so an unrelated response cannot resolve the wrong Promise.

Register temporary message and error listeners per request, remove them on completion, and enforce a timeout. If the worker crashes, terminate and clear the cached instance so the next attempt does not reuse a broken worker.

const worker = new Worker(
  new URL('../workers/ocr-accurate.worker.ts', import.meta.url),
  { type: 'module' },
);
const data = await file.arrayBuffer();

worker.postMessage({
  id: requestId,
  type: 'recognize',
  file: { data, type: file.type, name: file.name, size: file.size },
  language,
}, [data]);

Load and cache ONNX sessions once

Configure ONNX Runtime Web with same-origin WASM paths, sequential execution, graph optimization, and the WASM provider. Load three models and the character dictionary in parallel, then cache the shared Promise. The first request pays the model cost; later requests reuse sessions.

A single-thread configuration avoids cross-origin-isolation requirements and keeps memory behavior predictable. The tradeoff is lower peak throughput, which is acceptable for one image at a time.

const [det, cls, rec, dictionary] = await Promise.all([
  ort.InferenceSession.create('/models/ocr/det-mobile.onnx', options),
  ort.InferenceSession.create('/models/ocr/cls.onnx', options),
  ort.InferenceSession.create('/models/ocr/rec-unified-mobile.onnx', options),
  fetch('/models/ocr/rec-unified-dict.txt').then(response => response.text()),
]);

const boxes = await detectTextBoxes(det, sourceImage);
for (const box of boxes) {
  const crop = cropCanvas(sourceImage, box);
  const oriented = await classifyTextOrientation(cls, crop);
  blocks.push(await recognizeTextCrop(rec, oriented.canvas, dictionary, language));
}
return mergeRecognizedBlocks(blocks, language);

Preprocessing must match each model

createImageBitmap decodes the Blob, and OffscreenCanvas keeps pixel work in the worker. Reject images over a pixel limit before allocating more canvases. Detection resizes the longest side and rounds dimensions to a multiple of 32; classification stretches to its fixed input; recognition preserves aspect ratio and pads on the right.

Detection uses ImageNet normalization, while classification and recognition use Paddle-style normalization. Percentile contrast stretching improves faint text crops, but only when the detected contrast range is meaningful.

• Close ImageBitmap immediately after drawing it to OffscreenCanvas.
• Use CHW Float32 tensors with explicit RGB channel planes.
• Keep recognition height fixed and clamp maximum line width.
• Fill padded regions white rather than leaving transparent black pixels.

Post-processing is as important as inference

A raw detection probability map is not a list of lines. Thresholding, dilation, connected-component search, score filtering, padding, line-fragment merging, and reading-order sorting determine whether the recognizer sees complete text or broken pieces.

The recognizer output also needs careful decoding. Ignore the blank class, collapse consecutive identical classes, accumulate confidence, and optionally prefer characters from the selected language when their score remains close to the global best. This reduces cross-script noise without hard-blocking punctuation and Latin text.

Expose progress and limitations honestly

Report model, prepare, detect, classify, recognize, and merge as separate stages. During per-box work, include processed and total counts. This is more useful than one spinner because a page with many text regions can spend most of its time after detection.

Axis-aligned boxes and a 0/180 classifier do not fully solve perspective distortion, curved text, vertical writing, handwriting, or 90-degree rotation. When document-grade accuracy matters, add geometric rectification, broader orientation handling, or a specialized service.

Key takeaways

Browser OCR succeeds when model inference and classical post-processing are designed together. Move the work into a Worker, cache the three sessions, match preprocessing to each model, transfer large buffers, rebuild boxes and reading order, and present the remaining limitations clearly.