When 436,000 Messages Broke Our WebSocket Relay

At 11:40 on a weekday, the WebSocket relay behind a real-time odds and live-score platform we built was serving two connected clients and quietly building a backlog it would take the better part of a day to work through. Nothing in the infrastructure had changed. No traffic spike, no bad deploy, no dependency outage. The relay had simply been asked to say hello to a new client the way it always did — and that hello turned out to be 436,000 messages long.

This is the story of that meltdown: a bulk dump-on-connect that flooded a socket, a disconnect storm that fed on itself, and a backlog of over 3.4 million queued messages that starved the live data behind it. It's also the more useful story underneath — why "sync everything on connect" is a trap almost every real-time system falls into once, why backpressure isn't optional, and why adding more sockets never fixes a design bug.

§ 01The bulk dump that seemed reasonable

The relay sat between a database of live odds and every client that needed to display them. Its job was simple in principle: whenever a row changed, broadcast the change. The complication was the cold-start problem — a client that just connected has no state at all, so it needs the entire current picture before incremental updates mean anything.

The fix that shipped first was the obvious one. On every new connection, walk the full set of "upcoming" rows within a lookahead window and send each one as its own message:

// naive: dump full state to every new client
function onConnect(socket) {
  const rows = db.findMany({
    where: { startsAt: { gte: now, lte: fortyEightHoursOut } }
  });
  for (const row of rows) {
    for (const line of row.priceLines) {
      socket.send(JSON.stringify(line)); // one message per line item
    }
  }
}

It worked, in the sense that it shipped and passed review. Across a 48-hour lookahead window with dozens of markets per event and multiple providers per market, the "full current picture" fanned out to roughly 436,000 individual JSON messages per connection. Every reconnect replayed the entire dump from scratch. Nobody had put a number on "the full picture" before it was built — it was a phrase, not a budget.

§ 02Anatomy of the collapse

A WebSocket send doesn't fail loudly when the receiver can't keep up. It buffers. The server-side socket has an outbound buffer, the OS has a TCP send buffer, and ws.send() keeps accepting calls and stacking bytes behind them as long as you keep calling it. Nothing in that loop tells you to stop unless you're checking for it — and the dump-on-connect code wasn't checking for anything. It just called send() 436,000 times in a tight loop and moved on.

Here's what actually happened, in order:

A client connected. The relay began firing 436,000 messages at it as fast as the event loop would allow.
The client — or a proxy in between — couldn't drain the buffer fast enough. Somewhere between a few thousand and a few hundred thousand messages in, the connection died with close code 1006 — an abnormal closure, no close frame, the kind of failure that just means "the connection is gone" without saying why.
The client's reconnect logic had no backoff and no jitter. It reconnected roughly every 25–30 seconds, which is just how long it took to notice the drop and retry.
Reconnecting triggered the exact same 436,000-message dump, on a socket that was already struggling. The cycle repeated.
The relay's outbound queue was strictly FIFO. Every replayed dump piled in ahead of genuinely new data. The backlog crossed 3.4 million queued messages before anyone measured it directly — nobody had wired up queue-depth metrics, because nobody expected a queue to need one.

The socket wasn't dying from bad network. It was dying from its own payload.

— from the incident notes

The most damaging part wasn't the flood itself — it was what the flood did to everything queued behind it. Real, current price changes were arriving from upstream providers the whole time. Each one got appended to the same outbound queue as the stale replay traffic. A backlog measured in millions of messages, processed strictly in order, means a fresh update sent at minute one might not reach a client until hours later. The system wasn't just slow. It was confidently showing old data as if it were live.

§ 03Diff, don't dump

The fix was to delete the bulk sync entirely and replace it with an incremental one. Instead of pushing the entire current state on every connect, the relay kept a checkpoint timestamp and polled the underlying Postgres tables every five seconds for whatever had changed since that checkpoint:

// incremental diff, not bulk dump
let checkpoint = new Date(Date.now() - FIVE_MINUTES);

async function watchDatabaseChanges() {
  const changed = await db.findMany({
    where: { updatedAt: { gt: checkpoint } },
    take: 200,
    orderBy: { updatedAt: 'asc' }
  });
  for (const row of changed) broadcast(row);
  if (changed.length) checkpoint = changed.at(-1).updatedAt;
}

setInterval(watchDatabaseChanges, 5_000);

The result was a new connection receiving a couple hundred messages instead of 436,000, and the ongoing stream settling into small, regular diffs every five seconds rather than one enormous burst. No more 1006 storms, no more backlog, and — this mattered as much as anything — a live feed that was actually live again.

One detail is easy to get wrong here and worth calling out: the checkpoint has to start slightly in the past on boot, not at "now." Initialize it to the current instant and you create a silent gap — anything that changed in the window between the old connection dying and the new one starting is invisible forever, because the query only ever looks forward from the moment it began watching. A diff loop is only correct if its starting point accounts for the time it wasn't running.

Bulk sync answers "what is the entire state," which grows without bound as your dataset grows.
A diff loop answers "what changed since I last checked," which stays roughly constant regardless of dataset size.
The second question is the one your clients actually need answered on every update after the first.

§ 04Backpressure is not optional

The deeper lesson is that backpressure has to be a deliberate design decision, not something you find out you needed after a 1006 storm. Any time a producer can generate data faster than a consumer can absorb it, you need an explicit policy for what happens when the gap grows — not a hope that it won't.

In Node, the signal is right there if you look for it: ws.send() returns a boolean, and the socket exposes bufferedAmount. Ignore both and you'll never notice a slow client until memory graphs turn into a hockey stick.

// bounded outbound queue with a drop policy
function send(socket, msg) {
  if (socket.bufferedAmount > MAX_BUFFER_BYTES) {
    queue.dropOldest(socket.id);
    metrics.increment('ws_backpressure_drop_total');
    return;
  }
  socket.send(msg);
}

Three practical rules came out of this incident:

Cap per-connection burst size. Nothing should ever be allowed to enqueue hundreds of thousands of messages for a single socket in one pass, full stop — treat any code path that could do that as a bug by construction.
Pick a drop policy before you need one. Drop-oldest, drop-lowest-priority, or disconnect the client — any of these beats an unbounded FIFO queue that quietly buries fresh data under stale replay traffic.
Instrument queue depth, not just connection count. "2 clients connected" told us nothing was wrong. Queue depth would have told us everything, days earlier — the same instinct we wrote up in observability for small teams applies directly here: measure the thing that actually predicts failure, not the thing that's easy to measure.

§ 05Reconnects need jitter

The 25–30 second reconnect cadence wasn't a coincidence — it was the client's default retry interval, applied uniformly, with no randomization. Every disconnected client came back on almost the same schedule, which meant every retry re-triggered the same bulk dump at almost the same moment, which meant the relay never got a clean window to catch up.

Disabling the bulk sync sends 100k+ fewer messages per connection. That floods stopped causing abnormal closures before the sync could even finish. Ongoing changes now sync through the five-second diff loop instead.
— paraphrased from the fix notes

A fixed-interval retry is a synchronization bug waiting for an excuse. The standard fix is exponential backoff with jitter: wait a base delay, add a random offset, double the base delay on each subsequent failure up to a ceiling, and always randomize so that clients that disconnected together don't reconnect together. Without the random offset, a server recovering from an outage gets hit by the exact same thundering herd that caused the outage in the first place, just delayed by one retry cycle.

§ 06Why "just add more sockets" doesn't help

The instinctive scaling move when a relay is struggling is to add more relay instances and spread connections across them. In this case that would have made things measurably worse, not better.

Every new relay instance still ran the same bulk dump-on-connect logic against the same database, so horizontal scaling would have multiplied the flood by the number of instances, not divided it.
Fan-out patterns like this — one source of truth broadcasting to N relay nodes, each of which broadcasts to M clients — amplify a per-connection design flaw at every layer of the fan-out, which is exactly why libraries built around this shape (Socket.io's adapter model is the common example) put so much emphasis on getting the broadcast primitive right before scaling node count.
More instances also means more concurrent connections hammering the database for the same "full current state" query at connect time, which turns an application-level bottleneck into a database-level one.

Adding capacity is the right move when you're bottlenecked on volume. It does nothing when you're bottlenecked on a per-connection algorithm that does O(n) work proportional to total dataset size instead of O(changes) work proportional to what actually happened since the last check. Scaling out a bad algorithm just gets you the same failure in more places at once.

§ 07What we'd check before shipping the next one

The checklist that came out of this, for any WebSocket relay or fan-out service since:

Never let "send full state on connect" be the default. If a client genuinely needs a cold-start snapshot, cap it, paginate it, or serve it over a plain HTTP request instead of a message-per-row WebSocket blast.
Put a hard ceiling on messages-per-connection-per-second and alert when anything approaches it, not just when it's exceeded.
Treat abnormal closures as a queue-health signal first, network-health signal second. A 1006 storm is usually downstream of something the server is doing, not something the network is doing to it.
Always jitter your reconnect intervals, on both the client and anything else that auto-reconnects on your behalf.
Diff-based sync (poll for what changed since a checkpoint) scales with your rate of change. Full-state sync scales with your total dataset size. Pick the one that matches how your data actually grows.

None of this required new infrastructure. It required deleting a function that seemed reasonable when it was written, and replacing "send everything" with "send what changed." The five-second diff loop that replaced the flood is, on any given day, sending a tiny fraction of the volume the old code sent on a single connection — and unlike the old code, it never stops keeping up.

— End of essay. If your real-time system is starting to feel like this one did, let's talk before it gets to 3.4 million. Start a project →

When our WebSocket relay drowned in its own sync