Repository navigation
Add native Harmony chat and sequential tool calling - #973
Sayan Shaw (sayanshaw24) wants to merge 23 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1a563a06-04d5-47cf-9b20-af502d60b1eb
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1a563a06-04d5-47cf-9b20-af502d60b1eb
Build on the original Harmony PR protocol-selection idea while keeping regular tool calling unchanged. Qualify native token/template contracts, decode channels and handoffs, protect transcript data, and reuse shared admission and recovery. Generation is sequential; coding-model qualification remains blocked. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1a563a06-04d5-47cf-9b20-af502d60b1eb
The native implementation supersedes the original four-file accumulator prototype. Keep its commits and authorship without retaining a second Harmony parser. This merge deliberately leaves the new implementation tree unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1a563a06-04d5-47cf-9b20-af502d60b1eb
Carry per-call encoding outside public JSON through response publication, storage, replay snapshots and compaction. Keep raw-envelope callers compatible using the same metadata map. Recovered streaming items use the completed turn context so their encoding matches final responses and history. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1a563a06-04d5-47cf-9b20-af502d60b1eb
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1a563a06-04d5-47cf-9b20-af502d60b1eb
Qualify the GPT-OSS thinking-only tool-call shape and allow runtimes with terminal-token identity to use Harmony through the Generator backend. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Qualified Harmony models can enter the unsupported classic Generator path without required Engine terminal-token metadata, while decoder and native call behavior lack checked-in regression tests.
2 open issues
What changed in this PR
Adds native GPT-OSS Harmony support to Foundry Local’s chat pipeline, preserving tool-body encoding across hosting and Responses replay. The PR also expands predictive SDK sessions, embedding usage reporting, and regression coverage.
Changes:
- Adds token-ID-based Harmony decoding, protected prompt construction, native guidance, and runtime capability checks.
- Exposes ranking and typed-decision sessions in C++ and Python, with runtime lifetime and cancellation handling.
- Updates embedding token accounting, hosting reliability, documentation, tests, and CI model staging.
| File | Description |
|---|---|
sdk_v2/rust/tests/http_extract_test.rs |
Consolidates extraction checks and relocates test artifacts. |
sdk_v2/python/test/unit/test_non_generative_sessions.py |
Tests predictive contracts, validation, and handle cleanup. |
sdk_v2/python/test/unit/test_model_download_cancellation.py |
Parameterizes download cancellation coverage. |
sdk_v2/python/test/unit/test_imports.py |
Checks new predictive API exports. |
sdk_v2/python/test/integration/test_non_generative_sessions.py |
Adds native ranking and decision integration tests. |
sdk_v2/python/test/conftest.py |
Adds cached predictive-model fixtures. |
sdk_v2/python/src/foundry_local_sdk/session.py |
Implements typed ranking and decision sessions. |
sdk_v2/python/src/foundry_local_sdk/session_types.py |
Defines predictive request and response types. |
sdk_v2/python/src/foundry_local_sdk/__init__.py |
Exports predictive sessions and types. |
sdk_v2/python/README.md |
Documents predictive APIs and lifecycle. |
sdk_v2/python/examples/ranking_and_decision.py |
Demonstrates predictive session usage. |
sdk_v2/js/test/openai-client-dispose.test.ts |
Tests request-scoped session release. |
sdk_v2/INFERENCING.md |
Documents image input formats and limitations. |
sdk_v2/cs/test/FoundryLocal.Tests/ManagerLifetimeTests.cs |
Consolidates lease tests with bounded waits. |
sdk_v2/cpp/test/sdk_api/shared_test_env.h |
Allows explicit embedding-model selection. |
sdk_v2/cpp/test/sdk_api/responses_vision_test.cc |
Exercises standard image data URLs. |
sdk_v2/cpp/test/sdk_api/non_generative_test.cc |
Adds predictive SDK integration coverage. |
sdk_v2/cpp/test/sdk_api/embeddings_test.cc |
Tests usage accounting and cancellation isolation. |
sdk_v2/cpp/test/internal_api/web_service_test.cc |
Checks predictive session closure preserves loaded models. |
sdk_v2/cpp/test/internal_api/non_generative_runtime_test.cc |
Tests predictive queuing, shutdown, and dispatch. |
sdk_v2/cpp/test/internal_api/non_generative_contract_test.cc |
Tests embedded-NUL rejection. |
sdk_v2/cpp/test/internal_api/ipc_channel_test.cc |
Covers socket startup and unlink races. |
sdk_v2/cpp/test/internal_api/host_lifetime_test.cc |
Consolidates reconnect-deadline coverage. |
sdk_v2/cpp/test/internal_api/bootstrap_descriptor_test.cc |
Synchronizes owner-removal assertions. |
sdk_v2/cpp/test/internal_api/azure_catalog_test.cc |
Tests catalog metadata fallback and precedence. |
sdk_v2/cpp/test/CMakeLists.txt |
Registers predictive tests and IPC instrumentation. |
sdk_v2/cpp/src/service/responses_handler.cc |
Stores generated tool-call encodings. |
sdk_v2/cpp/src/service/non_generative_handlers.cc |
Returns HTTP 503 for runtime capacity exhaustion. |
sdk_v2/cpp/src/service/embeddings_handler.cc |
Returns measured embedding token usage. |
sdk_v2/cpp/src/model.h |
Declares loaded predictive-runtime acquisition. |
sdk_v2/cpp/src/model.cc |
Adds predictive runtime acquisition and I/O descriptors. |
sdk_v2/cpp/src/manager.cc |
Drains predictive requests during destruction. |
sdk_v2/cpp/src/inferencing/session/types.h |
Adds native raw and JSON call encodings. |
sdk_v2/cpp/src/inferencing/session/session.h |
Adds request-lifetime and model-binding hooks. |
sdk_v2/cpp/src/inferencing/session/session.cc |
Dispatches predictive sessions and acquires request leases. |
sdk_v2/cpp/src/inferencing/predictive/session_executor.h |
Extracts bounded, cancellation-aware runtime execution. |
sdk_v2/cpp/src/inferencing/predictive/non_generative_session.h |
Declares predictive session lifecycle state. |
sdk_v2/cpp/src/inferencing/predictive/non_generative_session.cc |
Implements predictive JSON dispatch and runtime leases. |
sdk_v2/cpp/src/inferencing/predictive/non_generative_runtime.h |
Adds runtime interfaces and capacity errors. |
sdk_v2/cpp/src/inferencing/predictive/non_generative_runtime.cc |
Uses the cancellation-aware executor. |
sdk_v2/cpp/src/inferencing/model_load_manager.h |
Declares predictive lifecycle coordination. |
sdk_v2/cpp/src/inferencing/model_load_manager.cc |
Implements predictive leases and shutdown draining. |
sdk_v2/cpp/src/inferencing/generative/toolcalling/tool_call_context.h |
Tracks Harmony protocol and body encoding. |
sdk_v2/cpp/src/inferencing/generative/toolcalling/grammar.cc |
Builds native Harmony guidance. |
sdk_v2/cpp/src/inferencing/generative/openresponses/response_store.h |
Stores per-call encoding metadata. |
sdk_v2/cpp/src/inferencing/generative/openresponses/response_store.cc |
Preserves encoding metadata through storage and replay. |
sdk_v2/cpp/src/inferencing/generative/openresponses/response_converter.cc |
Restores call encodings and forwards reasoning effort. |
sdk_v2/cpp/src/inferencing/generative/openresponses/response_chain.h |
Carries encoding metadata in replay chains. |
sdk_v2/cpp/src/inferencing/generative/genai_model_instance.h |
Exposes qualified Harmony capabilities. |
sdk_v2/cpp/src/inferencing/generative/genai_model_instance.cc |
Qualifies Harmony tokenizer, template, and runtime support. |
sdk_v2/cpp/src/inferencing/generative/embeddings/embeddings_session.h |
Threads usage accounting through embedding generation. |
sdk_v2/cpp/src/inferencing/generative/embeddings/embeddings_session.cc |
Counts prepared embedding input tokens. |
sdk_v2/cpp/src/inferencing/generative/chat/search_options.cc |
Enables Harmony guidance and host-side stops. |
sdk_v2/cpp/src/inferencing/generative/chat/prepared_chat_prompt.h |
Records Harmony continuation metadata. |
sdk_v2/cpp/src/inferencing/generative/chat/prepared_chat_prompt.cc |
Builds protected prompts and validates continuations. |
sdk_v2/cpp/src/inferencing/generative/chat/onnx_engine_chat_stream.cc |
Propagates terminal-token identity into turn usage. |
sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_engine.h |
Adds terminal-token metadata to results. |
sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_engine.cc |
Reads terminal-token identity from Engine events. |
sdk_v2/cpp/src/inferencing/generative/chat/harmony_protocol.h |
Defines Harmony controls and continuation state. |
sdk_v2/cpp/src/inferencing/generative/chat/harmony_decoder.h |
Declares decoder events and recovery interface. |
sdk_v2/cpp/src/inferencing/generative/chat/harmony_decoder.cc |
Parses native headers, bodies, and completion tokens. |
sdk_v2/cpp/src/inferencing/generative/chat/chat_transcript.h |
Clarifies native raw-body replay comments. |
sdk_v2/cpp/src/inferencing/generative/chat/chat_transcript.cc |
Clarifies custom argument normalization comments. |
sdk_v2/cpp/src/inferencing/generative/chat/chat_template.h |
Declares Harmony prompt projection. |
sdk_v2/cpp/src/inferencing/generative/chat/chat_template.cc |
Projects protected Harmony history and tool schemas. |
sdk_v2/cpp/src/inferencing/generative/chat/chat_generator.h |
Adds terminal-token identity to turn usage. |
sdk_v2/cpp/src/hosting/model_dispatcher.cc |
Adjusts hosted session creation and lock scope. |
sdk_v2/cpp/src/hosting/item_codec.cc |
Accepts native tool-call encodings over IPC. |
sdk_v2/cpp/src/hosting/ipc_channel_posix.cc |
Handles socket startup races before authentication. |
sdk_v2/cpp/src/contracts/non_generative.cc |
Rejects ranking answers containing embedded NULs. |
sdk_v2/cpp/src/contracts/embeddings.h |
Widens token counts to 64 bits. |
sdk_v2/cpp/src/contracts/chat_completions_converter.cc |
Forwards explicit reasoning effort. |
sdk_v2/cpp/src/catalog/azure_catalog_models.cc |
Adds capability-tag fallback and generic integer parsing. |
sdk_v2/cpp/src/api/hosted_manager.cc |
Reserves model selection during hosted session creation. |
sdk_v2/cpp/README.md |
Documents predictive sessions and native Harmony requirements. |
sdk_v2/cpp/include/foundry_local/foundry_local_cpp.inline.h |
Implements predictive JSON convenience methods. |
sdk_v2/cpp/include/foundry_local/foundry_local_cpp.h |
Exposes predictive sessions and handles GetMessage conflicts. |
sdk_v2/cpp/CMakeLists.txt |
Probes terminal-token support and includes new sources. |
.pipelines/templates/steps-build-linux.yml |
Stages and runs required predictive integration models. |
.pipelines/templates/stages-python.yml |
Stages and verifies predictive test packages. |
.pipelines/templates/fetch-models-from-artifacts-feed.yml |
Adds CLM and KEV artifact downloads. |
.github/instructions/cpp-tool-calling.instructions.md |
Documents Harmony token and completion invariants. |
.github/instructions/cpp-testing.instructions.md |
Documents embedding-model test overrides. |
Review effort: Balanced
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
67a8db7 addresses the additional review-summary finding: declarations and historical calls with whitespace or metadata separators in their names are rejected with INVALID_ARGUMENT before Harmony template rendering. Coverage exercises all six ASCII whitespace separators plus both internal metadata separator bytes. The leading-instruction and custom-input normalization fixes remain in place. Expanded combined native/Harmony selection passed 291 tests with one optional package skip, and the original supported Harmony HTTP matrix passed 14/14 again. Native recipient selection and protocol behavior for valid names are unchanged. |
There was a problem hiding this comment.
🔵 Needs a closer look
Human assessment is needed because transcript projection and stored-input replay can lose historical tool-call encoding distinctions, while the Harmony grammar branch lacks regression coverage.
No open issues 🎉
Review effort: Balanced
Carry known per-call encoding metadata through stored input replay as well as output replay. Cover native JSON versus raw history bodies, choice-specific headers, reserved boundary IDs, and native patch grammar admission. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Addressed the latest provenance/coverage concerns in 29d8b3e. Stored-input reconstruction now receives the same trusted per-call encoding map already used for generated output reconstruction. Tests round-trip structured, native-JSON, and native-raw custom bodies through stored input and stored output, verifying the historical encoding, tool kind, exact payload, and rendered raw-vs-JSON body even when the current declaration admits raw bodies. Added Harmony grammar tests for automatic/required/named/none choice with and without reasoning, both legal recipient header orders, authoritative boundary IDs and EOS exclusion, native stock-patch bodies, and explicit rejection of unsupported native custom formats. Final combined-stack suites passed 518 tests against both updated GenAI and the exact pinned 0.19.0 runtime (one optional package test skipped each). Public caller payloads do not acquire trusted provenance from client-supplied fields. |
|
Final post-fix combined-stack regression also passed all 14 supported native Harmony HTTP gates, including streaming/non-streaming required/named choice, continuation, serial calls, same-tool/different-arguments, and failed-result no-retry. The shared stack additionally passed 16/16 HTTP gates on each of three native-protocol configurations. No prompts/assertions were relaxed. Combined integration head 63d28ef is byte-identical to the tested SDK tree, and all test-owned model servers have been stopped. |
Historical generated and stored canonical/JSON/raw calls keep their original body encoding when declarations change. Only unannotated caller history infers the current declaration, preserving existing caller behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Follow-up 63f98e0 addresses the remaining historical-format distinction explicitly. The transcript now distinguishes recorded generated/stored encoding from unannotated caller history. Recorded canonical, native-JSON, and native-raw bodies retain their original encoding even if the current custom grammar is added or removed. Only unannotated caller-supplied calls continue inferring the current declaration, preserving existing caller behavior. The stored-input/output regression now covers all three historical encodings with and without a current raw-body grammar, and a separate regression protects unannotated caller behavior. Focused combined-stack tests pass 519 cases (one optional package skip). |
Incorporate the merged shared tool-validation prerequisite and current main without rewriting branch history. Rename the raw-envelope parameter and assistant-prefix local rejected by MSVC warnings-as-errors; preserve all protocol behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>


Summary
Adds native GPT-OSS Harmony chat and sequential tool calling. Targets
main; the shared validation prerequisite #1302 is now merged.The implementation separates model protocol selection from ordinary tool framing and replaces the prototype parser while preserving the original contributors' commits.
What changes
Related changes and runtime requirements
main.Validation
These are previously completed validations; updating the branch does not imply a new GPU qualification run.
Latest main and CI refresh
mainafter Share tool-call batch validation and JSON grammar #1302 landed, preserving the existing branch history.Scope
Sequential tool use. The upstream Harmony template renders one tool call per assistant message, so true parallel calls remain outside this PR.