Purge tombstoned metastore rows from the control plane

[em3s]

Summary

The metastore keeps every row it was ever told to delete. A metadata DELETE writes the row back with Active.INACTIVE, and JdbcHashLabel.scanStorage applies LIMIT before the active filter, so tombstones spend the budget a live scan needs. Past 1000 of them under one service, DatastoreTableReferences.whole() refuses and the control plane's own cleanup endpoints stop answering.

This adds a purge that reads a metastore table directly, hands back what it would delete, deletes exactly that, and can put it back. Stateless: candidates returns a PurgeSet, and execute and restore accept the same document unchanged, so the file an operator holds is both the plan and the backup.

Temporary. It targets a table the data plane is already abandoning and goes away with JDBC in Phase 4 of #347.

Closes #485

Plan

Created by claude code (opus 5)

• 1. actionbase.control.metastores.* config — the JDBC URI and table pairs the purge may open, credentials from the environment, resolved at startup
• 2. PK-cursor scanner — decode active and deleteTs with an empty field-name map, filter by service, report undecodable keys
• 3. PurgeSet as one type, serving as the candidates response and the request body of the other two
• 4. The three endpoints, rejecting a PurgeSet whose target is not in the configured list
• 5. Endpoint classification in ReadOnlyRequestFilter — execute and restore are writes
• 6. purge plan / show / apply / restore in the Go CLI
• 7. E2E test posting the candidates response back verbatim — resurrected tombstone, undecodable row, repeated execute, restore over a live key

Progress

• 1 — claude code (opus 5) (commit 5cfebd5)
• 2 — claude code (opus 5) (commit 8fe7c64)
• 3–7 — claude code (opus 5) (commit a02730f)
• Review round 1 fixes — claude code (opus 5) (commit db3c10d)
• Tests for the round 1 fixes, and the resolve comment corrected — claude code (opus 5) (commit 0fcd65f)

Changes

engine — metastore/purge/, standalone. MetastorePurge walks WHERE id > :cursor ORDER BY id, decodes two fields, and deletes on k and v together. PurgeFields reads the active flag off the front of the encoded value and the service off the front of the encoded key. PurgeTable is its own Exposed definition.

server — control/metastore/ holds the allowlist, the credentials and the service; api/control/metastore/ holds the three endpoints. candidates joins the read-only filter's allowed suffixes, execute and restore join CONTROL_WRITE_ENDPOINTS.

cli — purge plan | show | apply | restore, driving the endpoints over a file. apply and restore need --yes.

Three things worth naming

Nothing from the data model. The purge reads active and the service straight off the encodings with OrderedBytes and ValueUtils. No HashEdgeValue, no DecodedEdge, no EdgeSchema, no Graph. A tool that dies with JDBC should not be entangled with what is being retired around it, and it means no new dependency in server at all.

How old a tombstone is comes from update_ts, not the encoded delete_ts. Reaching that property means walking the whole property list, while the column says the same thing for a row that is currently inactive — the last write was the one that deactivated it — and says it in a WHERE clause against an index.

One document, three endpoints. candidates returns it and the other two take it back unchanged, so the CLI never reshapes anything. The rows therefore reach the client before anything is deleted, which is what makes the saved file a backup rather than a convenience. Committing a purge is keeping the file; reverting one is posting it to restore.

How to Test

./gradlew :engine:test --tests "*MetastorePurge*"
./gradlew :server:test --tests "*Metastore*" --tests "*ControlPurge*" --tests "*RequestFilterTest*"
cd cli && go test ./...

ControlPurgeE2ETest runs against a real table and posts the candidates response back byte for byte, which is the claim that has to keep holding.

Not touched

JdbcHashLabel.cad is broken — its deleteWhere lambda has k eq ... and v eq ... as separate statements, so Exposed takes the last as the predicate and the delete matches on value alone, ignoring the key. deleteOnLock is the caller. Out of scope here and it needs its own issue.

The root cause outlives this PR: scanStorage still cannot filter on active in SQL. An active column with an index would fix it properly, at the cost of a schema change and a backfill.

AI Assistance

• This PR was written largely with AI assistance.
  • Tool / model: Claude Code (Opus 5)