Measuring the parser — metrics, golden sets, parity scorecards, calibration, and eval discipline.
View All TagsMost sequence labelers throw away the parts of an address they don't understand, and the output format is built to hide it. We made ours hand every dropped character back, then aggregated them into a ranked list. The top of the list turned out to be the Polish alphabet.
Type 'Vienna, Austria' into our geocoder and it confidently returned a town of ten thousand people in Appalachia. The obvious fix was a retrain. We ran the diagnostic first, and the obvious fix turned out to be the wrong one.
Mailwoman couldn't find Tirana, or Yerevan, or 145 other capitals — not a wrong coordinate, silence. We were going to fix the parser. A ten-minute probe killed that plan, and a stale symlink nearly talked us into a GPU retrain to fix a model that was already fine. Two evals, two silent confounds, and the one dull habit that caught both.
You've got a hundred thousand addresses to reconcile. Two databases, the same clinics and providers scattered across both, each one spelled a dozen ways none.
We ran our 30 MB browser geocoder against the incumbents — Nominatim and Pelias — on real European addresses, scored the direct way. We'd won the US. Europe opened with a double-digit loss. The cause was silence: on messy addresses we returned no result at all. This post covers what caused the empty results, the three fixes that took us clear of both incumbents in Europe, and the one place — Australia — we've now diagnosed but not yet closed.
Our parser scores 28% on Spanish addresses by the metric that grades its labels. By the metric that grades where the address lands on Earth, it's within a couple of kilometers. Same model, same addresses. Only one of those numbers is measuring the thing we ship.
Overnight I taught the geocoder Canada. Or I thought I did. I pulled 843,000 Canadian postcodes, computed a centroid for every one, spot-checked the result — M5H 2N2, downtown Toronto, 43.652, −79.382, dead on — and validated the database every way I could read it: every postcode present, every coordinate right, and no other data disturbed. Green, top to bottom. This morning, before flipping it live, I asked the demo to find a Toronto address. It dropped the pin in Ohio.
We're trained to be suspicious of a score that looks too good. The scores that say you failed get a free pass — and this week three of them were lying, including one that had us reporting kilometers for a product that lands in meters.
We taught our parser the hardest street cases it gets wrong, the aggregate score climbed, and we almost shipped it. Then a held-out probe caught it forgetting what a town is.
Mailwoman ships two address parsers in the same box.