Skip to main content

Plan the migration of an existing data root

Outcome. You know which of the databases already under your data root can join the active selection as they are and which need a rebuild or a decision from you, and how the change would be undone.

The commands on this page are new and unreleased. They ship with the release that follows 10.7.0. mailwoman data migrate only plans in that release: it runs with --dry-run and refuses without it.

Prerequisites​

  • A data root that predates the active selection: databases under <data-root>/db/ placed by an earlier data pull or a local build, and no <data-root>/active.json.
  • No build or import writing to the data root while you plan. The plan reports a database with a WAL sidecar as blocked, because a writer may hold it open.

1. Print the plan​

bash
mailwoman data migrate --dry-run
mailwoman data migrate --dry-run --json > migration-plan.json

The plan writes no file. It asks the same readers the geocoder uses which file each one opens today: the candidate gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. through $MAILWOMAN_CANDIDATE_DB and its conventional path, each US state through releases.json and the legacy names, every published bundleevidence bundleThe pair of retrieval-augmented input channels (street-type + locality-surface) that feed lexicon membership as soft per-token evidence alongside the text. Shipped in 6.7.0; trained natively from step 0 in the from-scratch base line.'s conventional path, the OpenStreetMapOpenStreetMap (OSM). A community-curated global map database (ODbL-licensed) with addr:* tagged features and place hierarchies. A secondary corpus source and a source of street names. extractsextractA partial output file of the corpus build, written in Parquet format. The training pipeline streams extracts row by row. and the WOFWOF (Who's On First). An open-source gazetteer of places maintained by Mapzen/whosonfirst. Mailwoman builds a custom SQLite database from WOF GeoJSON repos, extended with postcode data, importance scores, and coincident-role relations. extractsextractA partial output file of the corpus build, written in Parquet format. The training pipeline streams extracts row by row..

2. Read each classification​

ClassificationWhat it meansWhat to do
importableThe file records a layer_manifest, and its role is knownLeave it. A migration registersinput modeThe Decision-A register switch: 'fragmented' (human-typed fragments — feeds the evidence channels) vs 'formatted' (complete records — runs the trained absence identity). Explicit on CLI/API; per-endpoint defaults (batch→formatted, autocomplete→fragmented); kind-derived otherwise. it in place
registry-manifestA published artifact with no layer_manifest; the registry's record states its roleLeave it. A migration writes its manifest from the registry record
source-rebuildA local build with no layer_manifest, so its inputs are unrecordedRebuild it from its source to record its inputs
unknown-provenanceNeither the file nor the registry records where it came fromKeep it where it is, or replace it with a published download
blocked-walA -wal sidecar is presentStop the writer, or seal the database, and plan again

Each line also reports a symlink and its target, a file with more than one hard link, and a path outside the data root, such as a gazetteergazetteerA geographical index that maps place names and postcodes to real-world coordinates. Mailwoman uses a custom-built Who's On First (WOF) SQLite database as its gazetteer — the 'atlas' half of the grammar/atlas architecture. pinned through $MAILWOMAN_CANDIDATE_DB. A migration registersinput modeThe Decision-A register switch: 'fragmented' (human-typed fragments — feeds the evidence channels) vs 'formatted' (complete records — runs the trained absence identity). Explicit on CLI/API; per-endpoint defaults (batch→formatted, autocomplete→fragmented); kind-derived otherwise. a file at the path the reader resolves today, so a symlink keeps pointing where it did.

A field the file does not record stays unrecorded. A migration never fills in a vintage, a license or a checksum the file lacks.

3. Check what changes for the geocoder​

A registered file is selected for the same role and scope its reader resolves today, so the geocoder reads the same bytes before and after. The plan copies 0 bytes. It hashes each registered file once, and the hashes line gives the total.

4. Know the rollback​

The plan's rollback line states how to undo it. With no earlier selection, the undo is to remove active.json and the installations/ directory. The registered files are neither moved nor changed. With an earlier selection, mailwoman data activate --rollback restores the generation the migration replaced.

Cleanup is a separate step​

Migration deletes no file. mailwoman data prune lists installed artifacts that no retained generation and no running session references. It keeps the active generation, the one before it and every generation a running session holds. Run mailwoman data prune --yes to delete what it listed. The command refuses when the selection changed after it planned. The layer installation reference states the retention policy in full.