fix(rate_limiter): enforce TPM pre-call in dynamic_rate_limiter_v3 priority pools

[devin-ai-integration]

TLDR

Problem this solves:

• Priority pool TPM was never enforced pre-call
• Model-wide TPM was equally unenforced in that hook
• A TPM-only model group enforced nothing at all

How it solves it:

• Zero increment no longer means "skip this counter"
• New CHECK_ONLY marker enforces a counter without advancing it
• TPM-only descriptors now reject at current >= limit

Relevant issues

Linear ticket

Resolves LIT-4800

Pre-Submission checklist

• I have added meaningful tests
• My PR passes all CI/CD checks (e.g., lint, format, unit tests)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both runs used the same proxy config (a 100 TPM model group, dynamic_rate_limiter_v3, priority_reservation set so the master key lands in the shared default pool at 50 TPM) and the same loop of six real gpt-4o-mini calls costing real $:

model_list:
  - model_name: gpt-4o-mini-priority
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
      tpm: 100

litellm_settings:
  callbacks: ["dynamic_rate_limiter_v3"]
  priority_reservation:
    prod: 0.6
    dev: 0.4
  priority_reservation_settings:
    default_priority: 0.5
    saturation_threshold: 0.4

general_settings:
  master_key: sk-1234

python litellm/proxy/proxy_cli.py --config config.yaml

for i in $(seq 1 6); do
  curl -s -w "http_status=%{http_code}\n" http://localhost:4000/v1/chat/completions \
    -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
    -d '{"model":"gpt-4o-mini-priority","messages":[{"role":"user","content":"Reply with the single word: ok"}],"max_tokens":5}'
done

Before, at 579f41d57f (base). Twelve calls, 180 tokens against a 100 TPM model group and a 50 TPM pool, nothing is ever refused:

--- request 1 ---   http_status=200   content: ok | total_tokens: 15
--- request 2 ---   http_status=200   content: ok | total_tokens: 15
--- request 3 ---   http_status=200   content: ok | total_tokens: 15
--- request 4 ---   http_status=200   content: ok | total_tokens: 15
--- request 5 ---   http_status=200   content: ok | total_tokens: 15
--- request 6 ---   http_status=200   content: ok | total_tokens: 15
(loop run a second time)
--- request 1 ---   http_status=200   content: ok | total_tokens: 15
--- request 2 ---   http_status=200   content: ok | total_tokens: 15
--- request 3 ---   http_status=200   content: ok | total_tokens: 15
--- request 4 ---   http_status=200   content: ok | total_tokens: 15
--- request 5 ---   http_status=200   content: ok | total_tokens: 15
--- request 6 ---   http_status=200   content: ok | total_tokens: 15

After, at 6735375de7 (this branch). The pool serves 60 tokens' worth of traffic and then refuses:

--- request 1 ---   http_status=200   content: ok | total_tokens: 15
--- request 2 ---   http_status=200   content: ok | total_tokens: 15
--- request 3 ---   http_status=200   content: ok | total_tokens: 15
--- request 4 ---   http_status=200   content: ok | total_tokens: 15
--- request 5 ---   http_status=429
error: {"message": "Priority-based rate limit exceeded. Model: gpt-4o-mini-priority, Priority: None, Rate limit type: tokens, Model TPM: 100, Model RPM: not configured, Remaining: 0, Model saturation: 60.0%", "type": "throttling_error", "code": "429"}
--- request 6 ---   http_status=429
error: {"message": "Priority-based rate limit exceeded. Model: gpt-4o-mini-priority, Priority: None, Rate limit type: tokens, Model TPM: 100, Model RPM: not configured, Remaining: 0, Model saturation: 60.0%", "type": "throttling_error", "code": "429"}

Type

🐛 Bug Fix

Changes

dynamic_rate_limiter_v3 sends {"requests": 1, "tokens": 0} into atomic_check_and_increment_by_n because a request's token count is unknowable before the model answers. _build_descriptor_atomic_payload then dropped every counter whose increment was non-positive, so the token key never reached the Lua KEYS list; no TPM limit was read, compared, or enforced. On a model group with tpm and no rpm the enforced descriptor set collapsed to zero keys and the pre-call hook returned OK with an empty status list.

The increment could not simply be relaxed to allow zero, because zero already means "do not track this dimension" for check_and_increment_tokens, which passes tokens only and relies on should_rate_limit for RPM; letting zero through there would double-enforce RPM. So the two meanings are now distinct values:

RATE_LIMIT_CHECK_ONLY: Final[Literal["check_only"]] = "check_only"   # litellm/constants.py
RateLimitIncrement = Union[int, Literal["check_only"]]

def _resolve_increment(raw):
    if raw == RATE_LIMIT_CHECK_ONLY:
        return 0      # enforce this counter, do not advance it
    return raw if raw > 0 else None   # None -> counter is not tracked at all

RATE_LIMIT_CHECK_ONLY counters reach the Lua script with increment = 0. Two adjustments make that a real check:

• the comparison uses max(increment, 1), so a check-only counter rejects at current >= limit exactly like RPM's increment of 1 does, rather than leaking the request sitting precisely at the reservation
• pass 2 skips the write, so the check leaves the counter and its window untouched

Token counters are written post-call by async_log_success_event, which increments the counter key and never touches the window key. A check-only counter therefore reads the raw counter value instead of treating a missing window as an empty one, matching how key/team/user TPM already reads through is_cache_list_over_limit; the counter's TTL is what bounds staleness. Both the Lua path and the in-memory fallback carry the same semantics.

Since both enforced descriptors carry tokens_per_unit, this restores model-wide TPM enforcement in this hook as well as the priority pool's.

One property is unchanged: tokens are only known post-call, so a simultaneous burst still passes the read together. The pre-call check enforces against prior completed usage, not a hard cap.

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/1a3869d0d9c64bdfa4c313091d87650e
Requested by: @shivamrawat1

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[greptile-apps]

Greptile Summary

This PR adds read-only TPM checks to dynamic rate-limiter priority and model-wide descriptors.

• Introduces a CHECK_ONLY increment marker and supports it in Redis Lua and in-memory atomic limiter paths.
• Preserves zero and negative numeric increments as untracked dimensions.
• Adds regression tests for TPM-only pools, rejection at the limit, read-only counter behavior, and RPM isolation.

Confidence Score: 3/5

This PR should not merge until CHECK_ONLY ignores token usage from an expired window instead of potentially returning a false 429.

The window and token keys have independent TTL timelines, while the new zero-increment branch deliberately reads the token counter even after the window expires, allowing completed usage from the prior window to block requests in the next one.

Files Needing Attention: litellm/proxy/hooks/parallel_request_limiter_v3.py

Important Files Changed

Filename	Overview
litellm/proxy/hooks/dynamic_rate_limiter_v3.py	Sends CHECK_ONLY for pre-call token enforcement while retaining post-call token accounting.
litellm/proxy/hooks/parallel_request_limiter_v3.py	Implements read-only atomic counters, but expired windows can still read prior-window token usage and falsely reject requests.
tests/test_litellm/proxy/hooks/test_dynamic_rate_limiter_v3.py	Adds coverage for priority-pool TPM rejection and non-mutating admission below the limit.
tests/test_litellm/proxy/hooks/test_parallel_request_limiter_v3.py	Covers CHECK_ONLY payload construction, boundary enforcement, and isolation from RPM checks, but not window/counter TTL skew.

_{Reviews (1): Last reviewed commit: "fix(rate_limiter): enforce TPM pre-call ..." | Re-trigger Greptile}

[greptile-apps]

Expired windows retain token usage

When the window key expires before the independently updated token counter, this branch reads usage from the previous window instead of resetting it, causing the first request in the new window to receive a false 429.

[devin-ai-integration]

This is deliberate, and resetting on window expiry is exactly what made the ticketed bug possible.

The window key is only ever written by counters that advance, and token counters don't: they're written post-call by async_log_success_event through async_increment_tokens_with_ttl_preservation, which touches the counter key alone. Treating a missing or rolled window as "usage is zero" therefore discards every token recorded so far, which is how a pool sitting at 100k tokens against a 180 TPM reservation sailed through unenforced.

Staleness is bounded by the counter's own TTL, which is window_size and is preserved rather than refreshed on increments, so a token counter cannot outlive one window. That is the same contract the key/team/user TPM path has always run on; is_cache_list_over_limit reads the raw counter value and never resets it against a window key either. So this branch matches the enforcement semantics already in production for every other TPM dimension, rather than inventing a new one.

[greptile-apps]

You're right, and I missed the asymmetry between who writes the window key and who writes the counter key.

My concern assumed both keys are always co-written, so a missing window → counter reset was safe. But for check-only (TPM) counters that's inverted: the counter is written post-call by async_log_success_event while the window key is only ever written by advancing (RPM-style) counters. Resetting current_counter = 0 on a missing window therefore discards live token usage, which is precisely the bug this PR fixes.

Relying on the counter's own TTL (window_size, preserved by async_increment_tokens_with_ttl_preservation) to bound staleness is the correct contract here, and it matches what is_cache_list_over_limit already does for key/team/user TPM. I'm satisfied with the approach.

[greptile-apps]

Shared sentinel lives inline

CHECK_ONLY is now imported by another hook but is defined inside this limiter implementation rather than constants.py, coupling consumers to the implementation module and bypassing the repository convention for shared sentinel values.

Rule Used: What: Require sentinel-like variables (e.g., `_NEG... (source)

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[codecov]

Codecov Report

❌ Patch coverage is 91.30435% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/hooks/parallel_request_limiter_v3.py	88.88%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed-hq]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_fix_priority_tpm_enforcement (6735375) with litellm_internal_staging (2d6b574)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (fc5ab31) during the generation of this report, so 2d6b574 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩