· By John Kavanagh

Processing Large Files in Node.js with Stream Backpressure

Photograph of water flowing over a moss-covered rock in a garden stream.
Image by Owen Hartley, licensed from Unsplash.

A catalogue conversion script can look perfectly respectable until someone gives it the real catalogue. Reading a file, parsing its records, transforming them, and writing the result is straightforward. Keeping all four representations in memory is an expensive interpretation of straightforward.

The first version is often a readFile() followed by a few array methods. For a small configuration file, I'd leave it alone. For an export that keeps growing, I want the converter to start writing before it has finished reading. I also want a slow destination to stop the source getting too far ahead.

That second requirement is backpressure. Merely using something called a stream doesn't establish it. The connection between the parts has to preserve it, and our own code must avoid quietly collecting everything in an array along the way.


Give the Consumer a Vote

Consider a file containing one JSON object per line. The source reads bytes, a decoder produces records, a transform changes each record, and a file stream writes the result. Each stage can be ready at a different time.

A writable stream's write() method returning false means its buffer has reached its threshold. The producer should stop writing until drain. It doesn't mean the write failed, and it certainly doesn't mean "try the same chunk again". That would produce a fairly creative catalogue.

When streams are connected properly, Node manages that conversation. Node 10's pipeline() gives us a callback for completion and forwards errors through the connected streams:

const { pipeline } = require('stream');
const fs = require('fs');

pipeline(
  fs.createReadStream(input, { highWaterMark: 4096 }),
  decodeRecords(),
  transformRecords(),
  fs.createWriteStream(temporary, { flags: 'wx' }),
  error => finishConversion(error)
);

The decoder, transform, and completion handler are application code here. pipeline() does not know our record format or decide whether a partially written catalogue is suitable for publication. It handles the stream connection, leaving those decisions with us.

These are Node's file and stream APIs, so this is also a concrete example of the distinction I covered in my guide to the browser and Node environments. A browser bundler won't turn a local file stream into a browser API.


A Chunk is Not a Record

The file reader can split a line anywhere. It can also split the bytes belonging to a single UTF-8 character. Calling chunk.toString() independently on every chunk and immediately parsing the result gives the decoder two opportunities to damage perfectly good input.

The example uses StringDecoder to retain incomplete characters, then keeps the unfinished line between calls. Complete lines are parsed in order. At the end, the decoder flushes any remaining character bytes and processes a final line without a newline.

There is a limit on record size too. Streaming a ten‑gigabyte file does not help if that file contains one ten‑gigabyte JSON value which the parser must hold together. This decoder rejects records larger than one mebibyte. That is an application limit, chosen for the example, and should be changed deliberately for a different format.

highWaterMark is a buffering threshold, not a process‑wide memory allowance. An object‑mode buffer counts objects; one object might contain a short label or an enormous description. The decoder can also emit several records from one incoming chunk. Its small input chunks and record‑size check matter alongside its object‑buffer setting.

I keep the transform similarly unambitious: one record in, one record out. An unbounded fan‑out into hundreds of asynchronous requests would introduce another queue, with another limit to design.


Don't Release the Next Record Too Early

An asynchronous transform must call its callback when that piece of work is finished. Calling it immediately, then completing the work later, tells the stream that there is room for more records even when there isn't.

The deliberately slow transform in the lab follows this shape:

transform(record, encoding, next) {
  let text;
  try {
    text = JSON.stringify(convert(record)) + '\n';
  } catch (error) {
    return next(error);
  }

  setTimeout(() => {
    next(null, text);
  }, delay);
}

The delay makes pressure visible; it isn't an attempt to simulate a particular disk. Because the callback stays outstanding, the transform does not accept the next record for processing straight away. Buffers further upstream fill to their thresholds, and the file reader eventually pauses.

This is one reason the distinction in my article on callbacks and promises matters beyond syntax. A promise‑returning function handed to an API expecting a callback does not automatically connect the two completion mechanisms. Somebody still has to tell the stream when the operation finished, including when it failed. That includes a conversion which throws: catch the error and pass it to next(error), so the cleanup code gets its turn too.


Compare the Same Input and Output

My file‑pipeline lab includes a deterministic catalogue generator and two processors. The eager version reads and transforms the whole file before writing. The streaming version connects the stages and starts producing output as records arrive.

Both discard the same padding field, preserve record order, and uppercase the same labels. The comparison hashes their output before considering memory or elapsed time. A low‑memory script which loses the final hundred records has not won anything useful.

Run each processor in its own process, against the same generated file, then repeat with a larger catalogue. Record peak resident memory as well as elapsed time. Sampling can miss short‑lived peaks, and resident memory includes more than JavaScript objects, so those readings need their runtime and sampling method beside them.

I would expect an eager implementation's retained input and arrays to grow with the catalogue. A bounded pipeline should show a different trend. I wouldn't promise a fixed memory figure, or that streaming always finishes faster. The slow consumer is meant to reveal where work waits, and waiting correctly is part of the result.


Keep Failure Away from the Finished File

The converter writes to a uniquely named temporary file beside the destination. Only a successful pipeline replaces the final file. On failure, it closes the file stream and removes the temporary output, leaving the previous complete version alone.

Putting both files in the same directory avoids a cross‑filesystem rename. This is still a small local‑file example: it does not claim power‑loss durability, coordinate competing writers, or make a remote filesystem behave like a local one.

The acceptance checks split input into single‑byte chunks, delay consumption, inject source and consumer failures, and feed the decoder malformed and oversized records. They also cancel a conversion after it starts. Cancellation must reach the streams still doing work; destroying a source which has already finished reading is too late to stop its buffered records being processed.

Successful completion is only half the contract. After every failed conversion, the previous output should still be complete, and the discarded temporary file should be gone. Once that holds, a larger catalogue becomes a longer job rather than an invitation to buy more memory.

Postscript

November 2026: Modern Node provides pipeline() through node:stream/promises, with an AbortSignal option for cancellation. My download now includes that version alongside the callback example. The current checks and memory comparisons use Node 22.23.5. The stream boundaries and output‑cleanup decisions still need the same care, whichever completion API you choose.

Downloadable Resource

  • Node.js File Pipeline Lab

    Compare eager and streaming NDJSON conversion, measure memory trends, and test slow consumers, cancellation, and preservation of complete output.

    Download ZIP

Looking for technical direction?

I support teams that need senior judgement on React, Next.js, headless CMS architecture, performance, migrations, and technical SEO.