[Sayan Shaw (sayanshaw24)]

Summary

Honor the loaded Generator implementation's support for host-driven prefill chunking, independently of tool calling.

Extracted from #1296 into a separate PR targeting main:

• Detect the optional GenAI capability with a compile-and-link feature probe.
• Query per-model support before injecting an automatic chunk size.
• Preserve explicitly positive model chunk sizes.
• Keep legacy defaults with older headers/libraries, including pinned GenAI 0.19.0, and report that compatibility choice during configuration.
• Add a regression for models that do not support external chunking.

This changes Generator prefill policy, not tool schemas, parsing, continuation history, sampling defaults, or CUDA graph capture.

Runtime requirement

The generic per-model query requires microsoft/onnxruntime-genai#2934 and a package containing that API. On an older runtime this PR deliberately preserves existing defaults; it does not claim to fix Bolt on the older runtime.

Related work

#1312 is the existing Bolt-specific alternative to suppressing the automatic chunk size without adding a capability API. Coordinate the choice rather than treating both approaches as mandatory dependencies. #1315 separately preserves Bolt's configured greedy default.

Neither this PR nor those model-runtime policy fixes belong in the native tool-calling diff.

Validation

• Built the standalone main-based branch against actual pinned GenAI 0.19.0 headers and library.
• 75 focused search-option, sampling, parsing, and Engine tests passed, including the new unsupported-chunking regression.
• The configure probe correctly reported the missing capability on 0.19.0 and selected the compatibility path.
• The moved modern-runtime integration was previously exercised on the combined stack; this extraction does not claim a new GPU qualification.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
foundry-local	Ready	Preview	Nov 12, 2026 12:13am UTC

[Copilot]

🟢 Approval recommended

No active review issues were identified for the model-aware prefill chunking update.

No open issues 🎉

What changed in this PR

Updates the C++ SDK’s Generator prefill policy to honor per-model support for host-driven chunking, independently of tool calling.

Changes:

• Detects the optional GenAI capability API with a compile-and-link probe and queries supported runtimes before applying automatic chunk sizes.
• Preserves explicit positive chunk sizes and legacy defaults on older runtimes, including pinned GenAI 0.19.0.
• Adds an unsupported-chunking regression test and documents compatibility behavior.

File	Description
sdk_v2/​cpp/​test/​internal_api/​chat/​search_options_test.cc	Tests that unsupported models receive no automatic chunk size.
sdk_v2/​cpp/​src/​inferencing/​generative/​chat/​search_options.h	Adds the capability parameter to search-option application.
sdk_v2/​cpp/​src/​inferencing/​generative/​chat/​search_options.cc	Gates automatic chunk sizing on model support.
sdk_v2/​cpp/​src/​inferencing/​generative/​chat/​onnx_chat_generator.cc	Queries model support, with a legacy-runtime fallback.
sdk_v2/​cpp/​README.md	Documents capability detection and compatibility behavior.
sdk_v2/​cpp/​CMakeLists.txt	Probes API availability and exposes the result to compilation.

_{Review effort: Balanced}