Migrations not automatically happening

[edwardmp]

Hi!

We recently had a minor production incident caused because database schema was out of date. It happened after upgrading Ledger from v2.3.1 to v2.4.11.

It was solved by running ledger migrate manually. Is this intended behavior that one needs to always run migrations manually? IIRC in previous Ledger versions it was automatically done.

Thank you!

[formance-support[bot]]

Yes, this is intended. ledger serve doesn't run schema migrations on startup by default. It only does so if you set --auto-upgrade (env AUTO_UPGRADE=true), which defaults to false. Without it, migrations run through the separate ledger migrate command, exactly as you found.

This default didn't actually change between your versions. --auto-upgrade was already false in v2.3.1 and is still false in v2.4.11, and both gate startup migrations behind that same flag. If it felt automatic before, the earlier deployment most likely had AUTO_UPGRADE=true set, or the migration was being run for you (see below).

Two supported ways to avoid the manual step:

• **Set **AUTO_UPGRADE=true on your serve deployment so it self-migrates on boot. Note that with this flag serve starts the upgrade in the background and begins serving immediately, so it doesn't guarantee the schema is ready before the first request.
• Run ledger migrate as a discrete step before rolling out the new serve version (an init Job / pipeline step). This is what the Formance Operator does automatically: on every version change it runs a <service>-migration Job with the migrate command, then rolls serve. If you deploy with the operator you already get this; if you're on plain containers/Helm, replicating that ordering is the robust option.

Either way, serve exposes a storage-driver-up-to-date health check that reports unhealthy until the schema reaches the required version. Wiring it into your readiness probe keeps pods on an out-of-date schema out of rotation until the migration lands, which prevents this class of incident.