[Sayan Shaw (sayanshaw24)]

Why

Tool-calling correctness currently has strong individual gates, but no reproducible per-model/per-backend contract that combines direct ORT GenAI behavior, Foundry HTTP protocol fidelity, and a sustained real coding-agent workflow.

What

• Add tool_call_qualification.py, a versioned orchestrator for one model/backend.
• Compose the existing Foundry protocol gate and real GitHub Copilot CLI coding-agent gate.
• Optionally require a direct ORT GenAI benchmark_toolcalling.py end-to-end report.
• Require exact logical model/backend correlation and retain the consumed report with its byte size and SHA-256.
• Support repeated runs and the native, minlength, and oneof schema profiles.
• Record the repository commit, GPU inventory, explicit requirements, artifact SHA-256 hashes, and private child-artifact locations in qualification.json.
• Mark the requested backend as operator-provided and explicitly distinguish file fingerprints from verified runtime identity.
• Save each completed child result incrementally, retaining earlier results when later checks time out.
• Reject invalid direct-model/backend reports before launching inference gates.
• Give timed-out gates a bounded cleanup window, then stop the owned process group or Windows Job Object.
• Accept the same loopback localhost, IPv4, and IPv6 endpoints as the coding gate.
• Run all three offline gate self-tests in a lightweight Linux/Windows/macOS CI matrix.
• Keep raw prompts, tool arguments, and model output in the existing private local run directories.
• Support current Copilot CLI releases where COPILOT_OFFLINE supersedes the removed --no-remote-export flag.
• Apply the oneof diagnostic profile to the platform's actual shell tool (bash or powershell).
• Prevent the exact-byte fixture from wasting its output budget re-deriving Unicode encodings; exact tool payload and final-file checks remain authoritative and unchanged.

Validation

Offline/component:

• python3 -m py_compile on both manual gates
• python3 -m ruff check on both manual gates
• tool_call_qualification.py --self-test
• coding_agent_gate.py --self-test
• tool_call_gate.py --self-test

At 31b08bfd, current main is merged, Python compilation and Ruff pass, all 12 qualification self-checks pass (including a real owned-child timeout and partial-result persistence), and the coding-agent/protocol self-tests pass on Linux. The first Windows CI run exposed an overly strict graceful-cleanup marker in the timeout self-test; the follow-up verifies actual Windows child-process termination after Job Object cleanup. The latest CI matrix establishes Windows and macOS results, not local claims.

Real CUDA qualification on an NVIDIA H200 with Qwen 3.8 27B:

• 3 complete repetitions
• each repetition passed the full protocol gate
• each repetition passed real Copilot CLI workflows for native, minlength, and oneof
• 9/9 coding-agent workflows passed with exact source bytes, real edit execution, deliberate tool failure, failing tests, repair, and independent passing tests
• zero provider error events in the passing qualification
• direct ORT GenAI report: 29 cases, 100% function selection, 100% enum validity, 100% unknown-parameter rejection, 100% clean stopping, and 82.76% exact end-to-end correctness
• the cross-stack run used a reviewed 82% direct-model baseline; callers can keep the conservative 100% default or set a separately reviewed per-model threshold

The direct report is produced by the benchmark introduced in microsoft/onnxruntime-genai#2687. It remains optional so Foundry-only qualification is explicit rather than silently pretending to be cross-stack.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
foundry-local	Ready	Preview	Nov 12, 2026 12:13am UTC

[Copilot]

🟡 Changes recommended

Cross-stack verdicts need direct-report model identity validation and retained, fingerprinted evidence.

2 open issues

• Validate direct benchmark model identity against --model · New
• Retain and fingerprint consumed GenAI report · New

What changed in this PR

Adds versioned, per-model/per-backend tool-calling qualification by combining the existing Foundry protocol and coding-agent gates with optional direct ORT GenAI evidence.

Changes:

• Adds repeated qualification runs across schema profiles, with environment fingerprints and consolidated verdicts.
• Updates the coding-agent gate for current Copilot CLI releases, both shell tools, and clearer exact-byte fixture instructions.
• Documents the qualification command and Foundry-only versus cross-stack scope.

File	Description
sdk_v2/​cpp/​test/​manual/​tool_call_qualification.py	Adds qualification orchestration, report assessment, and artifact metadata.
sdk_v2/​cpp/​test/​manual/​coding_agent_gate.py	Updates CLI compatibility, shell schema handling, self-tests, and fixture instructions.
sdk_v2/​cpp/​README.md	Documents qualification usage and recorded evidence.

_{Review effort: Balanced}

[Baiju Meswani (baijumeswani)]

This PR adds one repeatable way to run the existing tool-calling checks and collect their results for a model and backend. It can include a direct GenAI report, runs the Foundry HTTP checks and real Copilot coding workflows, and saves a versioned result with evidence links and hashes.

The approach is useful: it reuses the existing gates, clearly separates Foundry-only results from cross-stack results, keeps raw prompts and outputs local, and preserves the exact-byte checks. All three offline self-tests passed (103 checks), along with Ruff, Python compilation, and whitespace checks. I did not repeat the real-model qualification.

Before approval, please fix the outer-timeout cleanup and preserve completed results when a later check fails. The other inline comments are smaller compatibility and design suggestions, not requests to change model or inference behavior.