diff --git a/docs/proxy/config_settings.md b/docs/proxy/config_settings.md index 6489a9acd..fda7d6112 100644 --- a/docs/proxy/config_settings.md +++ b/docs/proxy/config_settings.md @@ -1164,6 +1164,7 @@ router_settings: | LITELLM_DISABLE_REDACT_SECRETS | When set to "true", disables automatic redaction of secrets (API keys, tokens, credentials) from proxy log output. Secret redaction is enabled by default. | LITELLM_DISABLE_ACCESS_LOG_PATHS | Comma-separated list of exact request paths whose uvicorn access-log lines should be dropped (e.g. health checks, root probes, metrics scrapes that flood logs). Path is matched against the portion before any query string. Empty/unset disables filtering. | LITELLM_MIGRATION_DIR | Custom migrations directory for prisma migrations, used for baselining db in read-only file systems. +| LITELLM_SKIP_INDEX_MIGRATIONS | When set to "true", migrations that only create indexes are recorded as applied without running at startup, so a large table is not locked by an index build during an upgrade. Build the indexes yourself with CREATE INDEX CONCURRENTLY and unset the flag afterwards. See [Skip index builds on a large database](./prod.md#skip-index-builds-on-a-large-database). **Default is false** | LITELLM_HOSTED_UI | URL of the hosted UI for LiteLLM | LITELLM_UI_API_DOC_BASE_URL | Optional override for the API Reference base URL (used in sample code/docs) when the admin UI runs on a different host than the proxy. Defaults to `PROXY_BASE_URL` when unset. | LITELLM_UI_PATH | Path to directory for Admin UI files. Used when running with read-only filesystem (e.g., Kubernetes). Default is `/var/lib/litellm/ui` in Docker. diff --git a/docs/proxy/prod.md b/docs/proxy/prod.md index 664a34c2f..98d1819f0 100644 --- a/docs/proxy/prod.md +++ b/docs/proxy/prod.md @@ -465,6 +465,23 @@ How LiteLLM ships migrations: 3. When you upgrade to a new version of LiteLLM, the migration file is applied to the database. [See code](https://github.com/BerriAI/litellm/blob/52b35cd8093b9ad833987b24f494586a1e923209/litellm-proxy-extras/litellm_proxy_extras/utils.py#L42) +### Skip index builds on a large database + +Some releases ship a migration that only adds an index, for example on `LiteLLM_SpendLogs`. Postgres builds it with a plain `CREATE INDEX`, which blocks inserts into that table until the build finishes. On a small database that is seconds; on a spend log table with hundreds of millions of rows it can stall request logging for the whole upgrade + +Set `LITELLM_SKIP_INDEX_MIGRATIONS=true` on whatever runs your migrations (the proxy pod, or the migration job if you use `DISABLE_SCHEMA_UPDATE`) to skip those builds. Every pending migration that only creates indexes (a `DROP INDEX` alongside is fine) is recorded as applied without being run, and the startup log names each one together with the indexes it would have built. A migration that only drops indexes always runs, since a drop is not a build. Migrations that add tables or columns always run, so the proxy never starts against a schema it does not know. The flag does nothing on a brand-new database, where the index is built on an empty table anyway, and nothing under `--use_prisma_db_push` + +Then build the index yourself without blocking writes, using the index name from the log: + +```sql +CREATE INDEX CONCURRENTLY IF NOT EXISTS "LiteLLM_SpendLogs_api_key_startTime_idx" + ON "LiteLLM_SpendLogs" ("api_key", "startTime"); +``` + +Postgres does not allow `CONCURRENTLY` on a partitioned table. If `LiteLLM_SpendLogs` is partitioned, create the index `ON ONLY` the parent, build it `CONCURRENTLY` on each partition, and `ATTACH PARTITION` each partition index to the parent one + +Unset the flag once your indexes are in place. While it stays set, every index-only migration in future releases is skipped the same way, and the schema check that runs after migrations never recreates an index owned by an index-only migration, even one applied long ago. So a dropped index stays dropped until you rebuild it yourself or unset the flag. A skipped migration's own `DROP INDEX` statements are not run either, so an index it replaces stays in place until you drop it by hand + ### Read-only file system Running LiteLLM with `readOnlyRootFilesystem: true` is a Kubernetes security best practice that prevents container processes from writing to the root filesystem. LiteLLM fully supports this configuration.