[GPT-OSS] Strict tool call and constrained decoding for Harmony

[yzong-rh]

Purpose

Following #45003 and #45396, attempt to implement constrained decoding for tool calls for Harmony models.

Related: #35904. Also "embed content constraints in structural tags".

Problem

vLLM's constrained decoding for the Harmony delays enforcing response_format constraints (json_object, json_schema, structural_tag) until <|channel|>final...<|message|> is detected. This works correctly for JSON formats but creates a mismatch for structural-tag tool calling, since those constraints need to apply during the commentary/tool-call phase rather than only in the final-message channel.

Solution

Rewrite json_object/json_schema response_formats into a Harmony-aware structural_tag (encoding the analysis and final-message boundaries directly in the grammar) so that constrained decoding is governed entirely by the structural tag throughout the entire generation.

1. Define custom structural tags for GPT-OSS models that constrains the entire generation, not just the final channel

• Two paths: with and without builtin tools (guided vs fallback-guided)
• Without builtin tools, the constraints will ensure each generated function has 100% the correct format
• With builtin tools, the constraints take a fallback path that only ensures some tool is called (no validation on tool name or arguments).

2. Override HarmonyParser's adjust_request so that JSON response_format gets wrapped into structural tags that apply over the entire generation.
3. Update CC and Responses API to call adjust_request for Harmony models
4. Clean up GptReasoningParser's reasoning_end methods and tests (they are no longer required).

Test Plan

.venv/bin/python -m pytest \
  tests/parser/test_harmony.py \
  tests/reasoning/test_gptoss_reasoning_parser.py \
  tests/entrypoints/openai/parser/test_harmony_utils.py \
  tests/entrypoints/openai/parser/test_harmony_render_parity.py \
  tests/entrypoints/openai/responses/test_harmony.py \
  tests/entrypoints/openai/responses/test_harmony_utils.py \
  tests/entrypoints/openai/responses/test_response_input_to_harmony.py \
  tests/entrypoints/openai/responses/test_function_call_parsing.py \
  tests/entrypoints/openai/responses/test_mcp_tools.py \
  tests/entrypoints/openai/responses/test_serving_responses.py \
  tests/entrypoints/openai/chat_completion/test_serving_chat.py \
  tests/entrypoints/scale_out/derender/test_derender.py \
  tests/entrypoints/tool_parsers/test_openai_tool_parser.py \
  tests/entrypoints/unit_tests/test_context.py

BFCL Multiturn

Manual with:

Query: "What's NVIDIA's P/E ratio?"

Tools: {
  "type": "function",
  "name": "get_current_pe_ratio",
  "description": "Get the current price-to-earnings ratio for a stock ticker.",
  "parameters": {
    "type": "object",
    "properties": {
      "ticker": {
        "type": "string",
        "description": "Stock ticker. Example: INTC for Intel Corporation."
      }
    },
    "required": [
      "ticker"
    ],
    "additionalProperties": false
  }
}

Benchmarking

vllm serve openai/gpt-oss-20b   \
--port 8886   \
--enable-auto-tool-choice   \
--reasoning-parser openai_gptoss   \
--tool-call-parser openai   \
--tensor-parallel-size 2   \
--enable-expert-parallel   \
--no-enable-prefix-caching   
--moe-backend flashinfer_cutlass

vllm bench serve --model openai/gpt-oss-20b --port 8886   \
--backend openai-chat --endpoint /v1/chat/completions   \
--dataset-name hf   \
--dataset-path gorilla-llm/Berkeley-Function-Calling-Leaderboard   \
--bfcl-categories simple   \
--num-warmups 5   \
--temperature 0   \
--percentile-metrics ttft,tpot,itl,e2el     \
--max-concurrency 128  \
--num-prompts 500

Test Results:

429 passed, 6 skipped, 1 xfailed, 1 xpassed

Chat Completions

	multi_turn_base	multi_turn_long_context	multi_turn_miss_func	multi_turn_miss_param
gpt-oss-20b-baseline	35.50%	23.00%	29.50%	33.00%
gpt-oss-20b-guided	58.00%	32.50%	42.00%	46.00%
gpt-oss-20b-fallback-guided	55.00%	32.50%	40.00%	46.00%
gpt-oss-120b-baseline	62.50%	38.50%	53.00%	47.50%
gpt-oss-120b-guided	61.50%	38.00%	53.50%	53.50%
gpt-oss-120b-fallback-guided	62.50%	38.00%	53.00%	48.50%

Responses

	multi_turn_base	multi_turn_long_context	multi_turn_miss_func	multi_turn_miss_param
gpt-oss-20b-baseline	36.00%	18.50%	28.50%	30.00%
gpt-oss-20b-guided	55.50%	33.00%	44.50%	43.00%
gpt-oss-20b-fallback-guided	55.00%	32.50%	39.50%	45.00%
gpt-oss-120b-baseline	65.00%	39.50%	53.00%	49.50%
gpt-oss-120b-guided	63.00%	38.50%	50.00%	50.00%
gpt-oss-120b-fallback-guided	65.00%	39.50%	54.00%	49.50%

JSON object:
{
  "content": "{\"name\":\"functions.get_current_pe_ratio\",\"arguments\":\"{\\\"ticker\\\":\\\"NVDA\\\"}\"}",
  "reasoning": "We need to get current P/E ratio for NVIDIA. Use function get_current_pe_ratio with ticker NVDA.",
  "role": "assistant"
}

JSON schema:
{
  "content": "{\"company_name\":\"NVIDIA\",\"pe_ratio\":24.5}",
  "reasoning": "We need to get current P/E ratio for NVIDIA. Use function get_current_pe_ratio with ticker NVDA.",
  "role": "assistant"
}

Baseline:
============ Serving Benchmark Result ============
Successful requests:                     500       
Failed requests:                         0         
Maximum request concurrency:             128       
Benchmark duration (s):                  6.22      
Total input tokens:                      97006     
Total generated tokens:                  48360     
Request throughput (req/s):              80.37     
Output token throughput (tok/s):         7773.67   
Peak output token throughput (tok/s):    10127.00  
Peak concurrent requests:                270.00    
Total token throughput (tok/s):          23366.98  
---------------Time to First Token----------------
Mean TTFT (ms):                          146.05    
Median TTFT (ms):                        57.08     
P99 TTFT (ms):                           541.45    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          9.94      
Median TPOT (ms):                        10.30     
P99 TPOT (ms):                           12.76     
---------------Inter-token Latency----------------
Mean ITL (ms):                           11.75     
Median ITL (ms):                         7.79      
P99 ITL (ms):                            144.34    
----------------End-to-end Latency----------------
Mean E2EL (ms):                          1050.85   
Median E2EL (ms):                        830.60    
P99 E2EL (ms):                           4096.74   
==================================================

Guided:
============ Serving Benchmark Result ============
Successful requests:                     500       
Failed requests:                         0         
Maximum request concurrency:             128       
Benchmark duration (s):                  6.37      
Total input tokens:                      97006     
Total generated tokens:                  48147     
Request throughput (req/s):              78.55     
Output token throughput (tok/s):         7563.75   
Peak output token throughput (tok/s):    9246.00   
Peak concurrent requests:                259.00    
Total token throughput (tok/s):          22803.09  
---------------Time to First Token----------------
Mean TTFT (ms):                          166.89    
Median TTFT (ms):                        107.92    
P99 TTFT (ms):                           554.89    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          10.02     
Median TPOT (ms):                        10.38     
P99 TPOT (ms):                           12.50     
---------------Inter-token Latency----------------
Mean ITL (ms):                           12.76     
Median ITL (ms):                         8.54      
P99 ITL (ms):                            140.11    
----------------End-to-end Latency----------------
Mean E2EL (ms):                          1077.68   
Median E2EL (ms):                        875.18    
P99 E2EL (ms):                           4674.10   
==================================================

cc @bbrowning @chaunceyjiang

Used AI assistance

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[sfeng33]

We can define the tags directly in HarmonyParser

[yzong-rh]

Implemented complete structural tags in vLLM (inside HarmonyParser). Although still importing helpers from structural_tag_registry.py.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @yzong-rh.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[chaunceyjiang]

I would recommend implementing the complete structural tag directly in vLLM rather than relying on this kind of patch-based solution.

[yzong-rh]

Implemented complete structural tags in vLLM. Bypasses the bugs and better performance.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[yzong-rh]

The structural tags help gpt-oss-20b, which has trouble following the Harmony format, a lot.
20% uplift in BFCL multiturn-base if we decide the enable it be default (right now it's no enabled unless function strict=True).

[bbrowning]

The change looks good overall. The only concern I have left, that we discussed via chat, is if a client passes structured_outputs directly the new unconditional is_reasoning_end of True will cause that to start enforcing output immediately. I've had to fix bugs from users in the past to ensure we only start enforcing this once reasoning has ended, and I suspect we'll break some in the wild usage if they happen to fall through the case where they are passing structured_outputs param but not anything else that would cause us to construct the proper Harmony structural tag.

[yzong-rh]

If a client passes structured_outputs directly the new unconditional is_reasoning_end of True will cause that to start enforcing output immediately

Good catch. If a client passes structured_outputs directly, it will indeed start enforcing the output immediately because enable_in_reasoning is always true for GPT-OSS.

We can turn a user-passed request.structured_outputs into a reasoning-aware structural_tag as we do currently with request.response_format / request.text.format.

I didn't do this to mostly to avoid complexity. My reasoning was that structured_outputs is a vLLM specific internal field and we should keep it straight-forward and honest -- advanced users might want to constrained the entire model output directly using structured_output. However, current users of structured_outputs would see a behavior change.

Will update structured_outputs handling later to avoid the user-facing behavior change.

[arpera]

LGTM

[yzong-rh]

Rebased and added commit that "converts a user-passed request.structured_outputs into a reasoning-aware structural_tag".

Unit tests pass locally; BFCL guided looks good.

While testing, found an edge case where the final tags leak into the constrained output because the model generates them char by char instead of as a single special token e.g. < | e n d | > instead of <|end|>.

This isn't a Harmony specific bug however. Will create and issue for this later.

[bbrowning]

We talked about the constrained edge case and the tag leak over Slack, and I'm ok deferring solving that for now as it looks like a bit deeper wiring into Structural Tags here to have it wire those special tokens in as ids instead of text so that we don't end up generating special tokens via multiple subword regular tokens. Not something to forget, but a deeper fix into how we wire these structural tags into xgrammar.

The only other thing that stands out to me is this removes the only real implementation of prepare_structured_tag, leaving the only callsite calling into the no-op implementation. That can also be deferred for the sake of fixing this, but effectively becomes dead code we can cleanup unless there is future intent to use that for something. I'm ok deferring that because it is technically part of our reasoning parser interface today since it's defined on ReasoningParser, so that may need some wider consideration before removal.

Thanks for all the work cleaning this up for this family of models!

[yzong-rh]

tests/parser/test_harmony.py::TestAdjustRequest failed in CI due to unintentional mutation of VLLM_ENFORCE_STRICT_TOOL_CALLING in two other tests. Removed said side-effect.

[yzong-rh]

tests/parser/test_streaming.py::test_parse_delta_required_tool_choice_kimi_k2_ids broke because it silently relied on VLLM_ENFORCE_STRICT_TOOL_CALLING mutation from tests/parser/test_parse.py. Fixed both tests via monkey patch. Maybe I should extract these fixes into another PR.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @yzong-rh.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[hanahhh]

Thanks for driving this PR @yzong-rh. I'm interested in seeing this land. Some of our customers are interested in using this feature.
Would you be able to rebase onto latest main to unblock final review/merge? Happy to help with a small conflict-resolution commit or retest after rebase if useful.

[yzong-rh]

Hey hanahhh, was hoping to see this land first #49227. Will push on both to make this happen, hopefully by EOW.

[bbrowning]

I reviewed the latest changes and still good to merge this - thanks!

One thing I'll note for a potential future follow-up - I believe if tool_choice="auto", user supplies a response_format/structured_outputs in their request (like json schema), and at least one tool has strict set, we end up not constraining the final output if the model decides not to call a tool. I believe we'd need to take that user-supplied json schema into account when building our structural tag in that case so that it gets applied when guiding the final channel output. That's a few combinations of things to hit this, and I could be reading things wrong here so worth double-checking before trying to implement that. Either way, this change as-is gets things cleaned up nicely for the general cases where we need to guide outputs!