How model releases will be evaluated

Mailwoman's planned release policy will require a modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' to meet explicit quality requirements and improve an overall parsingaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. score. An upgrade may change individual answers, and some countries or address patterns may score lower than in the previous version.

This policy direction was adopted on 2026-10-20. Its evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. tooling is partly in the repository, and no numerical requirement has been registered. This page describes the intended procedure. Published releases have not been certified against it.

What an acceptable release means

A release specification will state the countries and address tasks the modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' must handle, the minimum acceptable scores, and the evidence needed to establish them. An address task might be reading a PO BoxPO boxA numbered mailbox at a post office used as a delivery address instead of a physical street location. Mailwoman tags it as the po_box component; structurally the same family as a subpremise., recovering an apartment number, or distinguishing a streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. name from an administrative name.

The candidate must meet every required minimum and exceed the published modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.''s overall score by the amount the specification requires. A higher average cannot compensate for failing a required task. The published modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' provides the comparison for gains and lossesloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss.. The specification defines acceptable quality.

Countries receive requirements based on the product's intended coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present.. Historical inclusion in a regression check gives a countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. no permanent privilege. A countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. with insufficient evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. evidence will be identified explicitly. If the release requires that countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head., the missing evidence prevents approval.

How parsing quality will be measured

The primary measure will be exact parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. accuracy: the fraction of evaluated addresses whose predicted component labelscomponent tagOne of the 25 labels in Mailwoman's address schema — street, locality, region, postcode, house_number, unit, po_box, country, venue, intersection, and others. Each parsed span carries exactly one component tag. and boundaries match the annotated answer completely. The scorer counts both missing and additional components as errors. It will document the normalization rules it applies.

Each countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. receives its own score. The overall score is the sum of those countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. scores divided by the number of countries in a fixed evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. set. This gives each countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. equal influence, regardless of its evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. address count.

The report will also show component-level results. These help distinguish failures involving postcodespostcodeThe country-specific postcode (US ZIP, French code postal, etc.). Mailwoman handles postcode parsing entirely by rule classifier — a regex problem, not an ML one., streetstreetThe named linear feature along which house numbers are ordered. Decomposes into a name plus street affixes; one of the Tier 2 fine labels. names, unitsunitA subdivision of a building — apartment, suite, floor — that refines a street address. Mailwoman's unit component; a designator plus identifier forms a subpremise. and other components. A geocodinggeocodingThe process of converting an address into geographic coordinates (latitude and longitude). Mailwoman geocodes in a multi-tier cascade: exact address-point match → street interpolation → locality centroid. Each tier is progressively coarser but more widely available. claim will require a separate evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. of returned coordinates against reference coordinates at a stated distance tolerance.

Why the evaluation data matters

The evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. collection will have its own version and membership, maintained independently of a candidate's trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. corpuscorpusThe BIO-labeled training data used to train Mailwoman's neural classifier. Assembled from real sources (OpenAddresses, National Address Database) and synthetic extracts (boundary stress, order variants, negative space). Managed by @mailwoman/corpus.. Both the candidate and the published modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' will be evaluated on the same inputs.

The report will identify the sources, sampled address counts and known overlap with trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. data. Related examples, such as several renderings of one address, contain shared information. The uncertainty calculation must account for that relationship.

Data used to choose a modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' will be separate from the data used to confirm the final release decision. A changed evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. collection will require a new comparison of both modelsneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'. Results from different collection versions will remain separately identified.

What uncertainty means for approval

An evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. estimates performance on the population its examples represent. A confidence interval describes the uncertainty around that estimate under the evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.'s sampling assumptions.

Approval will require enough evidence that every minimum is met and the required overall improvement is achieved. If the evidence establishes that a requirement is unmet, the candidate fails. If the available evidence cannot settle the question, the result is insufficient evidence. Both outcomes prevent approval.

A small sample can reveal a large failure. It may still be too small to establish acceptable quality or distinguish a small improvement. Reports will give the evaluated address counts and uncertainty alongside the scores.

Why answers can change

The policy permits a release to perform worse on some inputs while improving the declared overall score and satisfying every required minimum. Release reports will identify measured lossesloss functionA number measuring how wrong the model's predictions are on a batch of examples. Training minimizes it. Mailwoman's loss combines per-token negative log-likelihood with the CRF sequence loss. and explain the accepted tradeoffs.

Curated regression examples will continue to reveal failures that averages can conceal. A changed answer can lead to a focused evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. of the affected address pattern. Its effect on release approval will depend on the stated quality requirements.

Repeated trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. runs will help assess whether a trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. method produces consistent results. The particular exported modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' intended for publication must pass its own evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error.. Results averaged across several trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. runs cannot establish that an individual modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' meets the requirements.

For an application that depends on specific address patterns, evaluate an upgrade against representative inputs from that application. The release report describes the measured countries, tasks and datasets. Your traffic may have a different distribution.

Adoption status

As of 2026-10-20, the repository contains the following parts of the procedure. None of them has run end to end on a real evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. collection or modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'.

• An exact parseaddress parsingThe process of decomposing a free-text postal address string into structured components — house number, street name, locality, region, postcode, and country — so a geocoder can resolve them to coordinates. scorer that counts missing and additional components as errors, and a held-out evaluator that writes one paired outcome per address for each modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.'.
• A group identifier on related examples, versioned evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. dataset manifests, an audit of overlap with trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input. data, and a generator for a per-countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head. coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. registerinput modeThe Decision-A register switch: 'fragmented' (human-typed fragments — feeds the evidence channels) vs 'formatted' (complete records — runs the trained absence identity). Explicit on CLI/API; per-endpoint defaults (batch→formatted, autocomplete→fragmented); kind-derived otherwise..
• A versioned release specification and an evaluator that produces pass, fail or insufficient evidence from paired outcomes.
• A trainingtrainingThe process of adjusting a model's parameters so its predictions match labeled examples, by repeatedly measuring error and nudging the weights to reduce it. Distinct from inference, when the trained model is run on new input.-run grader that reads release eligibility from that evaluator's result for the exact modelneural classifierThe machine learning model at the core of Mailwoman's parser — a transformer encoder (~30M parameters) trained from scratch to do BIO token classification over addresses. It learns the 'grammar' of address formats; the gazetteer supplies the 'atlas.' file and specification version, and reports regression examples as findings rather than vetoes.
• Removal of the earlier rule that let any regression example in FR, US, DE or GB block a release.

The first specification version registersinput modeThe Decision-A register switch: 'fragmented' (human-typed fragments — feeds the evidence channels) vs 'formatted' (complete records — runs the trained absence identity). Explicit on CLI/API; per-endpoint defaults (batch→formatted, autocomplete→fragmented); kind-derived otherwise. no required countrycountryThe top-level address component (an ISO country). Closed-vocabulary, so it is best handled by a deterministic matcher feeding a proposal rather than a retrained model head., task, minimum, improvement amount or confidence level. Every evaluationevalRunning the model against a held-out golden dataset and computing per-component F1, exact-match, calibration, and resolved-coordinate error. against it therefore returns insufficient evidence, and the procedure cannot approve a release yet. The earlier promotion checksconditional promotionA release discipline where a model ships to production only if it passes pre-registered metrics (e.g. 'DE locality ≥ 70%, no tag regresses > 2pp'). Failing models can be uploaded as experiments but not promoted to default. still run until a specification version registersinput modeThe Decision-A register switch: 'fragmented' (human-typed fragments — feeds the evidence channels) vs 'formatted' (complete records — runs the trained absence identity). Explicit on CLI/API; per-endpoint defaults (batch→formatted, autocomplete→fragmented); kind-derived otherwise. their replacements. A migration record lists each earlier check and its status.

Related

• Locales and tiers describes the current coveragecoverageThe fraction of a population or region for which a data source has real, non-placeholder entries — e.g. 47% rooftop coverage on Texas addresses. Distinct from accuracy on the rows that are present. declarations.
• How to read our numbers explains the limits of benchmark results.
• Report a parsing bug describes the evidence needed to investigate a specific input.