We Cleaned 8,000 Records for Under Thirty Dollars

Every data-heavy product eventually inherits the same problem: thousands of records that were never meant to live in one schema. On a pet-nutrition platform we built, that meant roughly 8,000 product records pulled from an open product database export, a handful of manufacturer pages, and a stack of photographed labels — three sources that agree on almost nothing. Ingredient lists sit mid-sentence in French, German, and Spanish. Nutrient panels use different units for the same number. Some rows are barcodes with no text attached to them at all. None of it is malformed, exactly — it's just never been asked to sit in the same table before.

The honest version of this story is that we didn't hire a data-labeling firm and we didn't write a thousand-line rules engine. We wrote a small amount of deterministic code to do the boring 90%, pointed a cheap vision+text model at the narrow 10% that actually needed judgment, and tracked the API bill the whole way through. Total AI spend for normalizing, tagging, and extracting structured fields across the dataset: under $30. This is the playbook, including the parts of the dataset the model never managed to rescue.

§ 01What we actually had

"Messy" undersells it. A single ingredients field might read "protéines de volaille déshydratées, riz, maïs, graisse de poulet (conservée aux tocophérols mixtes)" — French, with a parenthetical that itself contains a comma-separated sub-list. A guaranteed-analysis panel scraped from a manufacturer's HTML might report protein as a range instead of a minimum, or label ash as "Total Minerals" because the product ships in a market that uses that convention. A photographed label might have the ingredients panel folded out of frame in one shot and back in frame in the next.

None of this is a single problem you solve once. It's a few dozen small, recurring inconsistencies, and the mistake we didn't want to make was routing all of them through one enormous do-everything prompt and hoping a big model would sort it out per record. That's the expensive way to burn a budget: paying frontier-model prices to re-derive the same "strip the French preamble" logic eight thousand times.

§ 02Don't pay a model to read junk

The first lever, and the one that did more for the bill than model choice ever did, was refusing to send anything to the API that a regular expression could already handle. Before a single token got billed, a deterministic parser stripped known preambles ("Ingredients:", "Ingrédients :", "Composition"), split on commas while respecting nested parentheses so a sub-list didn't get shredded, and dropped fragments that were mostly digits or OCR noise using a simple alphabetic-character ratio check.

That parser ran across all ~8,000 product descriptions and collapsed tens of thousands of raw ingredient mentions into a few thousand unique fragments — most pet food ingredient lists overlap heavily, so "chicken," "brewers rice," and "mixed tocopherols" show up in hundreds of products each. We then took only the top 500 most frequent unique fragments and sent those to the model in batches of 50, with a short pause between batches to stay clear of rate limits. Every fragment below that frequency threshold either got caught by a later cleanup pass or fell out as a rare edge case — a deliberate trade, not an oversight, because the marginal fragment at rank 501 covers a handful of products, and the marginal dollar spent normalizing it isn't worth it.

• Dedupe before you pay. The API doesn't care that "dried chicken" appeared 400 times — you only need to classify it once.
• Filter deterministically first. A cheap heuristic (alpha-character ratio, length, digit-only pattern) catches most OCR garbage without ever calling the model.
• Batch aggressively. 50 fragments per call, one JSON array back, instead of one call per fragment.

§ 03One small model, three narrow jobs

We used one Claude Haiku–class model for the entire pipeline, but never as a single general-purpose assistant — it ran three distinct, narrowly scoped prompts, each returning a fixed JSON shape.

The first job was ingredient normalization: given a batch of raw fragments, return a canonical snake_case name, a display name, a category, a quality tier, and a handful of booleans (is_named_protein, is_filler, is_artificial), plus a skip flag with a reason for anything that wasn't actually an ingredient — a nutritional claim, a stray percentage, unreadable OCR. Across roughly 240 batches this landed a 99.4% match rate on the resulting product-ingredient links, meaning fewer than 1 in 150 links came back without a resolvable canonical ingredient. The second job was tag mapping: classify each canonical ingredient into quality, concern, nutrition, processing, and diet-type tag groups from a fixed vocabulary, in batches of 80. That closed at 98.1% ingredient coverage. The third job was vision: send one to four photos of the same physical label — front, back, guaranteed-analysis panel, barcode close-up — and get back one structured object covering brand, life stage, guaranteed analysis by nutrient, the AAFCO statement, and a self-reported confidence level, so a partially blurry photo tells you it's low-confidence instead of quietly polluting the dataset with a wrong number.

The pattern underneath all three is the same: a fixed schema in the prompt, one job per call, and a parser on our side that validates every field's type before it touches the database — a number stays a number or becomes null, an enum field either matches one of the allowed values or gets discarded rather than trusted. We've written before about treating a model's output as a typed contract instead of a blob of text; the same discipline is what kept three cheap, narrow jobs from turning into one expensive, sloppy one.

§ 04Two tiers of matching: exact first, fuzzy when it's earned

Not every matching problem needs a model at all. Reconciling incoming product records against what's already in the database — the same job a Shopify catalog sync or a recall lookup needs — ran as a two-tier SQL cascade, cheapest and most confident tier first:

// tier 1 — exact barcode match, confidence 1.0
const hit = await sql`
  SELECT id, barcode FROM products WHERE barcode = ANY(${barcodes})
`;

// only what tier 1 missed falls through to tier 2
const unmatched = incoming.filter(p => !hit.has(p.barcode));

// tier 2 — fuzzy name match via pg_trgm, ranked by similarity
const [best] = await sql`
  SELECT id, similarity(name, ${title}) AS sim
  FROM products
  WHERE similarity(name, ${title}) > 0.3
  ORDER BY sim DESC LIMIT 1
`;

// sim > 0.9 auto-confirmed · 0.3–0.9 queued for human review
const confirmed = best?.sim > 0.9;

Barcodes either match exactly or they don't — there's no ambiguity to pay a model for. Only the records that miss tier 1 fall through to pg_trgm's trigram similarity, and even then the result isn't trusted blindly: above 0.9 similarity it's auto-confirmed, between 0.3 and 0.9 it's queued for a human to glance at, below 0.3 it's treated as no match. The same cascade shape shows up one layer down, matching raw ingredient text to canonical ingredients: exact match on canonical name, then display name, then the alias table, and only then a substring/token fuzzy fallback — gated behind a four-character minimum so a stray single-letter fragment like "A" doesn't fuzzy-match onto something absurd. Trigram similarity is the right tool here because the mismatches are typographic — "Grain-Free Chicken Recipe" vs. "Grainfree Chicken Recipe." When the mismatch is semantic instead — different words, same meaning — trigrams stop helping and you're in embedding-similarity territory, a different problem with a different cost profile.

§ 05The bill, and what it actually bought

The whole pipeline's AI spend came in between roughly $22 and $35, call it under $30 for 8,000+ records processed. Here's where it actually went:

• Ingredient normalization — ~240 batches, ~$5–7
• Tag mapping — ~30 batches, ~$1–2
• Guaranteed-analysis extraction from scraped manufacturer HTML — ~2,080 individual calls, ~$12–22
• Vision label scanning — 7 photo sets, ~$0.20
• Everything else (knowledge-base drafting, misc. classification) — ~$1–2

Guaranteed-analysis extraction was, on paper, the simplest of the three jobs — pull protein, fat, fiber, and moisture out of a scraped HTML snippet, one product at a time, as each manufacturer scraper ran. It's also the single biggest cost line in the whole project, bigger than normalization and tag mapping combined, for one unglamorous reason: it ran one call per product instead of fifty fragments per call. Batching, not model choice, was the dominant lever on cost — the vision job did the most visually impressive work in the pipeline and cost a fraction of a percent of the total budget, purely because there were seven calls instead of two thousand.

§ 06What the model couldn't rescue

The honest part of a cheap-curation write-up is reporting the coverage number you didn't hit, not just the match rates you did. Roughly 2,870 records had a barcode but no ingredient text at all — scraped from the open database with a gap in the source data itself. We tried enriching them by querying that same database's public lookup API directly, one barcode at a time. Five came back with usable ingredients. Sixty-two returned a 404. The other 2,803 simply had no ingredient text on the other end either — the API wasn't a richer source than the export, it was the same gap with a different door.

Four thousand, four hundred forty-three records don't get a quality score. Not because a model failed at the job — because the sentence it needed to read was never in the source.

No amount of prompt engineering fixes a field that was never populated upstream. So the scoring engine — the part of the product that actually grades a record 0 to 100 — simply refuses to score anything without linked ingredient text, rather than guessing from a brand name or category. About 4,443 of the ~8,000 products fall into that bucket today, and the product says so plainly instead of quietly assigning them a middling number nobody could defend. The scored subset also caps out well short of a perfect score by design — no record has ever cleared the mid-70s, because the scoring rubric doesn't hand out points for "unremarkable," only for ingredients and analysis it can actually back up.

That refusal to fabricate a number is the part of this pipeline we'd defend hardest. A model that's 99.4% accurate on the records it touches is only trustworthy if you're equally clear about the records it never touched. The full build — schema, scrapers, scoring engine, and all — lives at the pet-nutrition platform case study. The short version of the playbook, in order:

1. Filter deterministically before any API call — regex and heuristics handle most of the mess for free.
2. Deduplicate ruthlessly; classify the unique fragment once, not every occurrence.
3. Batch every call you can — it moved the bill more than any model swap would have.
4. Match exact-then-fuzzy, cheapest tier first, and gate fuzzy matches behind a confidence threshold.
5. Give one small model several narrow jobs instead of one job a vague, general prompt.
6. Measure and publish the real coverage number — and let the product refuse to score what it can't support.

— End of write-up. Sitting on a pile of records nobody's had time to clean? Start a project →