· By John Kavanagh

Bounded Concurrency for a Node.js CMS Import

Photograph of a row of shopping trolleys beside a supermarket wall at night.
Image by Liviu Florescu, licensed from Unsplash.

Promise.all() is quite happy to watch three thousand requests. The CMS at the other end may have a less accommodating view of the arrangement.

Once an import has moved beyond a handful of records, its control flow needs to answer more than "have the promises finished?" It needs to decide how many requests may be active, what happens when the server asks us to wait, and which records remain unresolved after a failure. Those decisions belong in the importer, rather than being left to whichever socket runs out first.

The example here reads a paginated catalogue and fills in its missing references. It uses Node.js 16.4.2 and the built‑in HTTP client against a local synthetic CMS. There are no real CMS credentials, and the requests only read data. I want the queueing and retry behaviour to be visible without accidentally importing anything into somebody's production environment.


Start Work When a Worker is Available

In my article on asynchronous forEach(), replacing it with Promise.all(items.map(load)) fixes one completion problem. It still starts every load() during the map() call. Wrapping the resulting promises in a limiter afterwards cannot unstart those requests.

Keep the source items waiting, and let a small number of workers claim them. The core loop is intentionally ordinary:

let cursor = 0;
const results = new Array(items.length);

async function worker() {
  while (cursor < items.length) {
    const index = cursor++;

    try {
      results[index] = {
        status: 'fulfilled',
        value: await load(items[index]),
      };
    } catch (error) {
      results[index] = {
        status: 'rejected',
        reason: error.message,
      };
    }
  }
}

await Promise.all(
  Array.from({ length: Math.min(limit, items.length) }, worker)
);

The index is claimed before the first await, so two workers in this JavaScript process do not claim the same array position. Each worker waits for its current item before taking another. There are only limit worker promises, however many items the page contains.

The results array uses the source index, not completion order. A slow reference therefore does not shuffle the final catalogue. The full implementation also retains the stable source ID beside each outcome and checks cancellation before starting another item.

This resembles the outcome policy in my guide to Promise.allSettled(), with an additional decision about when work starts. Changing the aggregate promise alone would not provide that limit.


A Concurrency Limit is Not a Rate Limit

Three requests at a time can still produce a great many requests per second if each completes quickly. A concurrency ceiling bounds simultaneous work. A service quota might instead count requests across a minute, an account, or several machines.

In the lab, a 429 response pauses subsequent attempts through one shared waiting point. Workers which are already in flight are allowed to settle, but the next attempt checks the pause before sending another request. Giving only the unlucky worker a sleep would leave its colleagues happily spending the same quota. Even a 429 on the last attempt sets the pause. The next item still has to wait.

The waiting point belongs to this importer process. It does not coordinate another process using the same credentials. If several jobs share a service‑wide quota, the limiter needs that wider scope too, or the jobs need an agreed allocation.

I keep retry waiting inside the worker's current item. That is conservative: the item continues to occupy its worker slot whilst waiting. It also prevents a growing side queue of deferred retries. A more sophisticated scheduler can improve utilisation, but it should earn the extra machinery with evidence of a real bottleneck.


Let the Server Influence the Retry

The HTTP definition of Retry-After permits either a number of seconds or an HTTP date. Treating both as a numeric delay works right up to the first server which chooses the other format.

For a date, calculate how far it is ahead of the current time. For a missing or unusable value, use the local retry policy. The example takes the longer of the server's hint and its own exponential backoff with a small random addition. It never shortens a valid server hint just to get back to work sooner.

A 503 can be retried under the same bounded attempt policy. A 400 is retained as a permanent item failure in this fixture. There are three attempts at most, and the operations are GET requests. Retrying a read is a different proposition from repeating a write whose response was lost. I would not reuse this policy for writes without knowing how the API prevents duplicate effects.

Random delay is useful for spreading retries, but random tests are rather less useful. The retry helper accepts a clock and random‑number function. Its unit checks use fixed values to prove the selected delay, including a date‑form server hint, without waiting for a real clock to catch up.


Keep Pagination from Growing Another Queue

The importer fetches one page, resolves that page's missing references, and then asks for the next page. It does not fetch the entire catalogue into an ever‑growing array of pending promises.

Pages can overlap. A set of stable IDs prevents a repeated record being resolved twice within this run, and the first occurrence determines output order. A second set tracks cursors. Seeing the same cursor again is an error, rather than a reason to request the same page until somebody notices the fan getting louder.

These sets and the final outcome report still grow with the number of records. Bounded active work does not mean constant total memory. For a much larger migration, I'd persist outcomes and track source identity in durable storage. The small fixture keeps them in memory so the concurrency policy remains easy to inspect.

The page itself has a size limit, and the HTTP adapter limits response size. Limiting workers whilst accepting an arbitrarily large page would move the memory problem into a different variable.


Stop Waiting When the Job is Cancelled

Cancellation reaches both active HTTP requests and retry timers. The HTTP adapter connects an AbortSignal to request.destroy(), and the waiting helper removes its abort listener when the timer settles. Workers check the signal again before claiming fresh work.

The report distinguishes fulfilled, rejected, and cancelled items. A repeated pagination cursor stops the traversal with the completed outcomes attached to the error. That gives the caller useful evidence without pretending an incomplete traversal succeeded.

My bounded‑import lab records active requests and attempts at the stub server. Its checks cover duplicate page contents, out‑of‑order completion, a permanent 400, retryable 429 and 503 responses, repeated cursors, and cancellation during active work. The accepted run produces one result for each of five unique IDs: four fulfilled and one rejected.

That report is deliberately modest. It describes one run; it is not a restart checkpoint or a claim that the same import can be resumed safely tomorrow. First make the requests and outcomes predictable. Durable recovery can then build on something more useful than a console full of "probably done".

Postscript

November 2026: Node 22 provides a built‑in fetch(), and my download now includes an adapter using it alongside the original HTTP client. The worker limit, shared pause, and retry decisions stay outside the adapter. Changing the transport does not change the service's quota, although it does give us one less bit of request plumbing to maintain.

Downloadable Resource

  • Bounded CMS Import Lab

    Exercise a bounded Node.js importer against a local CMS with duplicate pages, retry hints, transient failures, cancellation, and ordered outcomes.

    Download ZIP

Have a complex web platform issue?

Tell me what is blocked, what has changed, and what needs to be true after the fix. I'll come back with a practical next step.