How Do You Avoid Doubling Memory When Processing Large Files in a Browser?

Memory often doubles at a conversion boundary: old bytes remain reachable while a new string, pixel buffer or Worker copy is created. The useful optimization target is the number and size of simultaneously live representations, not the number of variables. Draw the path from input through processing, export and preview before changing code.

Inventory live representations

A 100 MiB input read into an ArrayBuffer and cloned to a Worker accounts for about 200 MiB of byte buffers. A Base64 version adds about 139.8 million characters (133.3 MiB if stored as one byte each). Actual string storage depends on the engine. A File may be disk-backed, so do not automatically count it as another full resident copy.

A small view can retain a huge backing buffer. If only a tiny header must survive, copying that header can release more memory than keeping a zero-copy view. Ownership and lifetime are as important as avoiding allocations.

Stream to a real output sink

This function copies bytes to a WritableStream, such as an already authorized file destination. It does not compress images. Any inserted transform must support incremental work and bound its own internal state. The destination must actually consume chunks instead of accumulating the entire result.

async function copyWithBackpressure(
  file: File,
  destination: WritableStream<Uint8Array>,
  signal: AbortSignal,
): Promise<void> {
  await file.stream().pipeTo(destination, { signal });
}

pipeTo applies backpressure and propagates completion and failure under its default options. Collecting chunks in an array and constructing a final Blob still retains the full output. new Response(stream).blob() also gathers the result. When a suitable file sink is unavailable, impose an output limit or offer a desktop workflow instead of promising constant memory.

Transfer ownership to a Worker

// Main thread: worker is an existing Worker.
const buffer = await file.arrayBuffer();
worker.postMessage({ buffer }, [buffer]);
// buffer.byteLength is now 0: ownership was transferred.
// Do not read or reuse views backed by buffer here.

The transferable is the ArrayBuffer rather than its typed-array view, and it must also appear in the message payload. After transfer, the sender cannot continue using the detached buffer. Keep task identifiers and progress in application state rather than a second input copy.

This example still reads the whole input. An incremental design can send one chunk and wait for acknowledgement before reading the next. Unbounded postMessage calls simply move the backlog into the message queue. Decoding, WASM memory and output buffers can still allocate inside the Worker, and libraries built around whole-document parsing need a different memory budget.

Bound concurrency and retained results

1. Start with one active task instead of Promise.all over all files; measure before increasing concurrency.
2. Write completed outputs promptly and retain status plus small thumbnails instead of every full Blob.
3. Release obsolete preview URLs, close unused ImageBitmaps and remove listeners or history entries that retain data.
4. On cancellation, stop reads and scheduling and discard late results from obsolete task IDs.

Chunk boundaries are not content boundaries. A UTF-8 character may span chunks, requiring a streaming decoder. A line reader also needs a maximum record size: a file with no newline can otherwise grow an unbounded partial-line buffer. Input streaming alone cannot fix that algorithmic choice.

Verify the improvement

Compare the same file and browser before and after the change. Record idle baseline, processing peak and post-cleanup baseline, then repeat ten cycles. Check output correctness and duration alongside memory. Include process memory because native image and Blob resources may not be fully represented in a JavaScript heap snapshot. A reduction is an empirical result, not a guarantee that usage will be exactly halved.