Eight feeds, one table

Eight data feeds, one event, and no two of them agreed on what to call it. That was the actual starting condition for a real-time odds and live-score platform we built: six feeds exposed genuine APIs, two didn't expose an API at all, and every one of the eight had its own idea of team identifiers, timestamp precision, and whether a match had already started. The product needed one clean, comparable table — a match, a market, an outcome, a price, per provider — updated on a schedule that tightened from hourly to once a minute as kickoff approached. Getting there had almost nothing to do with fetching data and almost everything to do with reconciling it.

This is a write-up of that reconciliation layer: the provider-adapter pattern, in anger, across eight sources that actively disagreed with each other on structure, identity, and truth.

§ 01Eight feeds, no agreement

Six of the eight sources had documented APIs — JSON over REST, one gzip-compressed, one served from a local mirror database instead of a live call. The other two had no API at all. For those, prices had to be synthesized from a primary feed using per-market, per-game correction coefficients derived empirically and written down in a spec — more on that shortly.

Collection frequency wasn't uniform either, and it couldn't be: hitting eight endpoints once a minute for every scheduled match for the next seven days would have been both wasteful and rate-limit suicide. The schedule tightened as a match approached:

• More than 6 hours out — every hour.
• 3 to 6 hours out — every 30 minutes.
• 1 to 3 hours out — every 15 minutes.
• 30 minutes to 1 hour out — every 5 minutes.
• Inside the last 30 minutes, and once live — every minute.

None of that is exotic on its own. The actual complexity was downstream of collection: normalizing eight structurally different payloads into one row shape without losing the information that made each source useful, and without letting any single provider's quirks leak into the schema the rest of the system depended on.

§ 02The adapter contract: one normalize(), eight implementations

The pattern that made this tractable is old and boring, which is exactly why it works: one adapter per feed, each implementing the same interface, each responsible for translating its provider's native shape into a single internal row format. Nothing downstream of the adapter layer is allowed to know that eight different sources exist.

// one adapter per feed, same shape out
interface FeedAdapter {
  provider: string;
  fetch(ref: ScheduleRef): Promise<RawPayload>;
  normalize(raw: RawPayload): OddsRow[];
  isLive(raw: RawPayload, now: Date): boolean;
}

interface OddsRow {
  matchId: string;       // our id, never theirs
  provider: string;
  market: string;
  outcome: string;
  price: number;
  isFallback: boolean;
  fetchedAt: Date;
}

This is the same discipline we wrote about in typing a system from day one: the interface is the contract, and every adapter is free to be as messy as its source requires internally, as long as normalize() hands back rows in the one shape everything else understands. One feed's payload nested a bet inside a tournament inside a category; another was flat key-value pairs; a third arrived gzip-compressed and needed decompression before it was even JSON. None of that mattered outside the adapter. What mattered was that eight normalize() implementations all produced the same OddsRow[], and that the rest of the pipeline — storage, staleness detection, fallback calculation, the WebSocket relay — was written against that one shape and never touched a provider-specific field again.

§ 03Whose ID is it anyway

Every feed had its own numeric ID for every team, and none of the eight ID spaces overlapped. One feed's team 117 was another feed's team 56869 was a third feed's plain string team code. There was no foreign key connecting any of them, because there was no shared authority — each provider had built its own internal taxonomy independently, for its own purposes, with no reason to agree with anyone else's.

The fix was a standing lookup table, not a runtime guess: one row per real-world team, one column per feed holding that feed's ID and display name for it, populated by matching on name similarity and reviewed as exact, fuzzy, or unmatched. A team that appeared as "NVISION" in one feed and "NOVAVISION" in another matched at roughly 0.82 similarity — high enough to link automatically, low enough that it got flagged for a human to confirm rather than trusted blindly. New teams appear constantly in esports; the table has to keep being fed, not built once and frozen.

Once that table exists, every adapter's normalize() step does one extra lookup — provider ID in, canonical team code out — and every downstream consumer works in canonical codes exclusively. The alternative, string-matching team names live on every request, is slower, less reliable, and turns a one-time data-quality problem into a recurring runtime one.

§ 04Faking an API: synthesizing two feeds from one

Two of the eight sources had no API at all — no endpoint, no feed, nothing to poll. Both mattered to the product anyway, so their odds were synthesized: take the current price from a primary feed, apply a documented per-market, per-game correction coefficient, and round.

The coefficients came from an offline statistical comparison — historical spreads between the primary feed and each absent provider, broken out by market type and by game, because the gap wasn't constant. A small pre-match correction on a match-winner market could triple once a game went live, because in-play pricing moves faster and the two synthesized providers lagged the primary feed by different amounts under volatility. The formula was simple; the constants weren't:

// correction coefficients derived offline, applied live
function applyCorrection(price, correctionPct) {
  const corrected = price * (1 + correctionPct / 100);
  return Math.round(corrected * 100) / 100; // two decimals, always
}
// e.g. primary = 2.00, correction = -4.2% -> 1.92

The other half was a fallback chain. The correction assumed the primary feed's price was current. When it wasn't — no update within 5-10 minutes pre-match, or 15 seconds to 2 minutes live — the base switched to a secondary feed instead, with the same coefficient table applied against it. Two synthesized providers, one real primary source, one real secondary as fallback, and a staleness detector deciding which real feed to trust. The two synthesized feeds never touched an API of their own, because they didn't have one — they were entirely a function of everyone else's data plus a documented policy.

§ 05The flag that lied

Several feeds exposed a boolean live field on every event, and for a while the system trusted it. That trust didn't survive contact with production. The flag flipped late relative to actual kickoff on some feeds, never flipped at all on feeds with no live endpoint behind it, and occasionally stayed true long after a match had genuinely ended. It looked authoritative and wasn't. The fix was to stop asking the provider and start asking the clock:

// don't trust the provider's flag — compute it
function resolveStatus(scheduledStart, now) {
  return now >= scheduledStart ? 'live' : 'prematch';
}

Time-based inference is crude by itself, so it was corroborated against an ancillary live-scores feed tracking real match state independently — matched to the pricing schedule by normalized team name within a rough time window and a similarity threshold. That corroboration wasn't a hard dependency; when it disagreed with the time-based guess, the time-based guess won, because a name match is itself probabilistic, not a guarantee. The lesson generalizes past odds feeds: any field a third party hands you describing its own state — isLive, isActive, status: "current" — is a claim, not a fact, and the real-time systems we've built since (including the relay work in the WebSocket backpressure piece) have all ended up independently verifying state rather than propagating whatever a producer says about itself.

§ 06Same match, eight names

Merging events across all eight feeds was a different problem from resolving team identity within one feed, and it needed a different key. Kickoff times drifted by several minutes between providers — some rounded to the nearest five minutes, some carried true-to-the-second timestamps that shifted slightly between a prematch listing and a live one — so an exact-timestamp match was too strict, and matching on team names alone risked collisions across unrelated fixtures on a busy day.

// merge the same real-world event across all eight feeds
function dedupeKey(match) {
  const [a, b] = [match.team1, match.team2]
    .map(normalizeTeamName)
    .sort();
  const hourBucket = floorToHour(match.startTime);
  return `${a}::${b}::${hourBucket}`;
}

function normalizeTeamName(name) {
  return name.toLowerCase().replace(/[^a-z0-9]/g, '');
}

Sorting the two team names before joining them absorbed home/away swaps between feeds that ordered participants differently. Bucketing the start time to the hour, rather than matching it exactly, absorbed the minutes of drift between providers without being loose enough to merge two different matches between the same two teams on the same day — which, in a sport with best-of series and rematches, is a real collision risk if you bucket too coarsely.

A second, narrower problem showed up inside a single feed: one provider occasionally issued multiple internal IDs for what was clearly one event, fragmenting predictions and odds across duplicates. That was solved separately, by scoring each duplicate on data completeness, picking the highest-scoring record as canonical, merging the rest into it, and marking those as duplicates everywhere downstream. Cross-feed merging and within-feed merging looked similar on the surface and needed almost entirely different logic — one reconciles identity across independent systems, the other is a single system's own data hygiene.

§ 07A provider ships v2: versioning the adapter, not patching it

Eventually one of the six API-backed feeds shipped a breaking change: a flat field structure where nested objects used to be, numeric price fields where stringified ones used to be, and a differently named timestamp field on a different epoch. Every downstream consumer of that adapter's output was still expecting the old shape, and dozens of active match schedules were mid-flight against the old endpoint when the new one went live.

The instinct to patch the existing adapter in place — branch internally on a version flag, handle both shapes in one file — is the wrong one for exactly the reason branching logic usually is: it turns one adapter into two adapters wearing the same trenchcoat, with shared state and untestable edge cases where the branches interact. What shipped instead was a second, fully independent adapter implementing the same FeedAdapter interface against the new API version, registered alongside the first under a version key. Both ran concurrently while schedules migrated one at a time from the old endpoint to the new one; the old adapter was retired only after nothing referenced it anymore.

Don't patch a live adapter to match a new API version. Ship a second one behind the same interface, migrate schedules across, and only delete the old path once nothing points at it.

The interface being narrow — fetch, normalize, tell me if it's live — is exactly what made this cheap. A new implementation of three methods is a small, reviewable, revertible change. A conditional buried inside an existing implementation is none of those things.

§ 08What carries over

None of the individual pieces here are unusual — adapters, a canonical identity table, time-bucketed deduplication, an anti-corruption layer in spirit if not in name. What matters is the order of priorities. Fetching data from eight sources is solved the day you write the eighth HTTP client. Agreeing on identity across those sources, deciding which one to believe when they disagree, and giving yourself a clean way to replace one without touching the other seven — that's the actual project, whether the eight sources are pricing feeds, inventory systems, or CRMs from four acquisitions. The adapter is the easy 20%. The normalization contract behind it is the 80% that decides whether the system holds together a year later.

— End of essay. If you're staring down a pile of disagreeing third-party feeds, we've done this before. Start a project →