From c31c177e8100dc36896ab16b05c71bf17d0a399c Mon Sep 17 00:00:00 2001 From: Yuneng Jiang Date: Thu, 3 Sep 2026 15:42:01 -0700 Subject: [PATCH 1/3] docs(proxy): document LITELLM_SKIP_INDEX_MIGRATIONS --- docs/proxy/config_settings.md | 1 + docs/proxy/prod.md | 17 +++++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/docs/proxy/config_settings.md b/docs/proxy/config_settings.md index 6489a9acd..ab032e925 100644 --- a/docs/proxy/config_settings.md +++ b/docs/proxy/config_settings.md @@ -1164,6 +1164,7 @@ router_settings: | LITELLM_DISABLE_REDACT_SECRETS | When set to "true", disables automatic redaction of secrets (API keys, tokens, credentials) from proxy log output. Secret redaction is enabled by default. | LITELLM_DISABLE_ACCESS_LOG_PATHS | Comma-separated list of exact request paths whose uvicorn access-log lines should be dropped (e.g. health checks, root probes, metrics scrapes that flood logs). Path is matched against the portion before any query string. Empty/unset disables filtering. | LITELLM_MIGRATION_DIR | Custom migrations directory for prisma migrations, used for baselining db in read-only file systems. +| LITELLM_SKIP_INDEX_MIGRATIONS | When set to "true", migrations made only of CREATE INDEX and DROP INDEX statements are recorded as applied without running at startup, so a large table is not locked by an index build during an upgrade. Build the indexes yourself with CREATE INDEX CONCURRENTLY and unset the flag afterwards. See [Skip index builds on a large database](./prod.md#skip-index-builds-on-a-large-database). **Default is false** | LITELLM_HOSTED_UI | URL of the hosted UI for LiteLLM | LITELLM_UI_API_DOC_BASE_URL | Optional override for the API Reference base URL (used in sample code/docs) when the admin UI runs on a different host than the proxy. Defaults to `PROXY_BASE_URL` when unset. | LITELLM_UI_PATH | Path to directory for Admin UI files. Used when running with read-only filesystem (e.g., Kubernetes). Default is `/var/lib/litellm/ui` in Docker. diff --git a/docs/proxy/prod.md b/docs/proxy/prod.md index 664a34c2f..64a7aebf7 100644 --- a/docs/proxy/prod.md +++ b/docs/proxy/prod.md @@ -465,6 +465,23 @@ How LiteLLM ships migrations: 3. When you upgrade to a new version of LiteLLM, the migration file is applied to the database. [See code](https://github.com/BerriAI/litellm/blob/52b35cd8093b9ad833987b24f494586a1e923209/litellm-proxy-extras/litellm_proxy_extras/utils.py#L42) +### Skip index builds on a large database + +Some releases ship a migration that only adds an index, for example on `LiteLLM_SpendLogs`. Postgres builds it with a plain `CREATE INDEX`, which blocks inserts into that table until the build finishes. On a small database that is seconds; on a spend log table with hundreds of millions of rows it can stall request logging for the whole upgrade + +Set `LITELLM_SKIP_INDEX_MIGRATIONS=true` on whatever runs your migrations (the proxy pod, or the migration job if you use `DISABLE_SCHEMA_UPDATE`) to skip those builds. Every pending migration made only of `CREATE INDEX` and `DROP INDEX` statements is recorded as applied without being run, and the startup log names each one together with the indexes it would have built. Migrations that add tables or columns always run, so the proxy never starts against a schema it does not know. The flag does nothing on a brand-new database, where the index is built on an empty table anyway, and nothing under `--use_prisma_db_push` + +Then build the index yourself without blocking writes, using the index name from the log: + +```sql +CREATE INDEX CONCURRENTLY IF NOT EXISTS "LiteLLM_SpendLogs_api_key_startTime_idx" + ON "LiteLLM_SpendLogs" ("api_key", "startTime"); +``` + +Postgres does not allow `CONCURRENTLY` on a partitioned table. If `LiteLLM_SpendLogs` is partitioned, create the index `ON ONLY` the parent, build it `CONCURRENTLY` on each partition, and `ATTACH PARTITION` each partition index to the parent one + +Unset the flag once your indexes are in place. While it stays set, every index-only migration in future releases is skipped the same way + ### Read-only file system Running LiteLLM with `readOnlyRootFilesystem: true` is a Kubernetes security best practice that prevents container processes from writing to the root filesystem. LiteLLM fully supports this configuration. From c749908ed6aa85bb75cfaf5cb5676efc4e1985f9 Mon Sep 17 00:00:00 2001 From: Yuneng Jiang Date: Thu, 3 Sep 2026 16:05:34 -0700 Subject: [PATCH 2/3] docs(proxy): say the post-migration schema check honors LITELLM_SKIP_INDEX_MIGRATIONS too --- docs/proxy/prod.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/proxy/prod.md b/docs/proxy/prod.md index 64a7aebf7..d13c11642 100644 --- a/docs/proxy/prod.md +++ b/docs/proxy/prod.md @@ -480,7 +480,7 @@ CREATE INDEX CONCURRENTLY IF NOT EXISTS "LiteLLM_SpendLogs_api_key_startTime_idx Postgres does not allow `CONCURRENTLY` on a partitioned table. If `LiteLLM_SpendLogs` is partitioned, create the index `ON ONLY` the parent, build it `CONCURRENTLY` on each partition, and `ATTACH PARTITION` each partition index to the parent one -Unset the flag once your indexes are in place. While it stays set, every index-only migration in future releases is skipped the same way +Unset the flag once your indexes are in place. While it stays set, every index-only migration in future releases is skipped the same way, and the schema check that runs after migrations never recreates an index owned by an index-only migration, even one applied long ago. So a dropped index stays dropped until you rebuild it yourself or unset the flag ### Read-only file system From 46812ed0510d515f970bf9717c9bd47c99e32574 Mon Sep 17 00:00:00 2001 From: Yuneng Jiang Date: Thu, 3 Sep 2026 16:25:10 -0700 Subject: [PATCH 3/3] docs(proxy): drop-only migrations always run under LITELLM_SKIP_INDEX_MIGRATIONS --- docs/proxy/config_settings.md | 2 +- docs/proxy/prod.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/proxy/config_settings.md b/docs/proxy/config_settings.md index ab032e925..fda7d6112 100644 --- a/docs/proxy/config_settings.md +++ b/docs/proxy/config_settings.md @@ -1164,7 +1164,7 @@ router_settings: | LITELLM_DISABLE_REDACT_SECRETS | When set to "true", disables automatic redaction of secrets (API keys, tokens, credentials) from proxy log output. Secret redaction is enabled by default. | LITELLM_DISABLE_ACCESS_LOG_PATHS | Comma-separated list of exact request paths whose uvicorn access-log lines should be dropped (e.g. health checks, root probes, metrics scrapes that flood logs). Path is matched against the portion before any query string. Empty/unset disables filtering. | LITELLM_MIGRATION_DIR | Custom migrations directory for prisma migrations, used for baselining db in read-only file systems. -| LITELLM_SKIP_INDEX_MIGRATIONS | When set to "true", migrations made only of CREATE INDEX and DROP INDEX statements are recorded as applied without running at startup, so a large table is not locked by an index build during an upgrade. Build the indexes yourself with CREATE INDEX CONCURRENTLY and unset the flag afterwards. See [Skip index builds on a large database](./prod.md#skip-index-builds-on-a-large-database). **Default is false** +| LITELLM_SKIP_INDEX_MIGRATIONS | When set to "true", migrations that only create indexes are recorded as applied without running at startup, so a large table is not locked by an index build during an upgrade. Build the indexes yourself with CREATE INDEX CONCURRENTLY and unset the flag afterwards. See [Skip index builds on a large database](./prod.md#skip-index-builds-on-a-large-database). **Default is false** | LITELLM_HOSTED_UI | URL of the hosted UI for LiteLLM | LITELLM_UI_API_DOC_BASE_URL | Optional override for the API Reference base URL (used in sample code/docs) when the admin UI runs on a different host than the proxy. Defaults to `PROXY_BASE_URL` when unset. | LITELLM_UI_PATH | Path to directory for Admin UI files. Used when running with read-only filesystem (e.g., Kubernetes). Default is `/var/lib/litellm/ui` in Docker. diff --git a/docs/proxy/prod.md b/docs/proxy/prod.md index d13c11642..98d1819f0 100644 --- a/docs/proxy/prod.md +++ b/docs/proxy/prod.md @@ -469,7 +469,7 @@ How LiteLLM ships migrations: Some releases ship a migration that only adds an index, for example on `LiteLLM_SpendLogs`. Postgres builds it with a plain `CREATE INDEX`, which blocks inserts into that table until the build finishes. On a small database that is seconds; on a spend log table with hundreds of millions of rows it can stall request logging for the whole upgrade -Set `LITELLM_SKIP_INDEX_MIGRATIONS=true` on whatever runs your migrations (the proxy pod, or the migration job if you use `DISABLE_SCHEMA_UPDATE`) to skip those builds. Every pending migration made only of `CREATE INDEX` and `DROP INDEX` statements is recorded as applied without being run, and the startup log names each one together with the indexes it would have built. Migrations that add tables or columns always run, so the proxy never starts against a schema it does not know. The flag does nothing on a brand-new database, where the index is built on an empty table anyway, and nothing under `--use_prisma_db_push` +Set `LITELLM_SKIP_INDEX_MIGRATIONS=true` on whatever runs your migrations (the proxy pod, or the migration job if you use `DISABLE_SCHEMA_UPDATE`) to skip those builds. Every pending migration that only creates indexes (a `DROP INDEX` alongside is fine) is recorded as applied without being run, and the startup log names each one together with the indexes it would have built. A migration that only drops indexes always runs, since a drop is not a build. Migrations that add tables or columns always run, so the proxy never starts against a schema it does not know. The flag does nothing on a brand-new database, where the index is built on an empty table anyway, and nothing under `--use_prisma_db_push` Then build the index yourself without blocking writes, using the index name from the log: @@ -480,7 +480,7 @@ CREATE INDEX CONCURRENTLY IF NOT EXISTS "LiteLLM_SpendLogs_api_key_startTime_idx Postgres does not allow `CONCURRENTLY` on a partitioned table. If `LiteLLM_SpendLogs` is partitioned, create the index `ON ONLY` the parent, build it `CONCURRENTLY` on each partition, and `ATTACH PARTITION` each partition index to the parent one -Unset the flag once your indexes are in place. While it stays set, every index-only migration in future releases is skipped the same way, and the schema check that runs after migrations never recreates an index owned by an index-only migration, even one applied long ago. So a dropped index stays dropped until you rebuild it yourself or unset the flag +Unset the flag once your indexes are in place. While it stays set, every index-only migration in future releases is skipped the same way, and the schema check that runs after migrations never recreates an index owned by an index-only migration, even one applied long ago. So a dropped index stays dropped until you rebuild it yourself or unset the flag. A skipped migration's own `DROP INDEX` statements are not run either, so an index it replaces stays in place until you drop it by hand ### Read-only file system