[Bug]: DeepSeek V3 and V3.1 streaming tool calls lose their arguments when the call ends in the last delta #21004

System Info

• TensorRT-LLM main at 687359e (tensorrt_llm/serve/tool_parser/deepseekv3_parser.py, deepseekv31_parser.py), pure Python. A CPU is enough to reproduce.
• Python 3.12
• Serving path: trtllm-serve ... --tool_parser deepseek_v3 or deepseek_v31, streaming chat completions with tools. Deltas carry several tokens with MTP or other speculative decoding, or with stream_interval above 1.

Who can help?

@NVIDIA/trt-llm-runtime-devs (CODEOWNERS for tensorrt_llm/serve)

Reproduction

from tensorrt_llm.serve.openai_protocol import ChatCompletionToolsParam
from tensorrt_llm.serve.tool_parser.deepseekv3_parser import DeepSeekV3Parser
from tensorrt_llm.serve.tool_parser.deepseekv31_parser import DeepSeekV31Parser

tools = [ChatCompletionToolsParam(function={
    "name": "get_weather", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}}})]

def stream(parser, deltas):
    calls = []
    for delta in deltas:
        calls += parser.parse_streaming_increment(delta, tools).calls
    calls += parser.finish(tools).calls  # what postprocess_handlers does when the request finishes
    return [c.name for c in calls if c.name], "".join(c.parameters for c in calls)

v3_head = ("<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_weather\n"
           '```json\n{"city": "Paris"}')
print(stream(DeepSeekV3Parser(), [v3_head, "\n```", "<｜tool▁call▁end｜>", "<｜tool▁calls▁end｜>"]))
# (['get_weather'], '{"city": "Paris"}')
print(stream(DeepSeekV3Parser(), [v3_head, "\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>"]))
# (['get_weather'], '')

print(stream(DeepSeekV31Parser(), ["<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_weather",
                                    "<｜tool▁sep｜>{}<｜tool▁call▁end｜><｜tool▁calls▁end｜>"]))
# (['get_weather'], '')

Expected behavior

The streamed tool call carries the same name and arguments as detect_and_parse returns for the same text, however the text is split into deltas.

actual behavior

When the last delta holds the closing fence and the end tokens (V3), or the separator, the arguments and the end tokens (V3.1, common for a call without arguments), the client receives the tool name with an empty arguments string, which is not valid JSON. A whole tool call in one delta gives the same result for both parsers.

additional notes

parse_streaming_increment sends at most one tool name or the arguments of one call per increment, and it sends the arguments one increment after the name. V3 sends the name only once the closing fence has arrived, so a stream whose last delta holds the fence ends after the name. V3.1 sends the name when the separator arrives. Neither class overrides BaseToolParser.finish, which returns nothing, so the arguments left in the buffer are never sent. postprocess_handlers.apply_tool_parser calls finish() once when the request finishes, which is the place to send them.

A related problem is out of scope here. When two whole calls arrive in one delta, the greedy first group of the streaming pattern gives the first call the arguments of the second. The fix I have leaves that case exactly as it is on main. It sends arguments in finish() only when the buffer holds one complete call whose name was already sent, and only the JSON object detect_and_parse reads for that call. I have it with tests ready and will open a PR that references this issue.