README.md

Foundry Local C and C++ SDK (v2)

The Foundry Local C++ SDK runs AI models locally through the Foundry Local engine. It provides a header-only, RAII-style C++17 wrapper over the stable C ABI for catalog access, model lifecycle management, chat, tool calling, embeddings, audio transcription, execution-provider management, and the optional OpenAI-compatible web service.

C and C++ are supported application interfaces to the same native runtime that powers the C#, JavaScript, Python, and Rust bindings. C applications use the public C API directly; C++ applications can use the C++ wrapper. Neither requires a separate server process.

The SDK implementation builds as C++20, while applications that consume the public wrapper need only C++17 or newer.

For an inference-focused introduction to models, sessions, requests, responses, items, and streaming across languages, see Inference with the Foundry Local SDK.

Features

• Model catalog -- discover model aliases, variants, cached models, and loaded models
• Model lifecycle -- download, load, unload, remove, and register local model assets
• Chat -- stateful chat sessions, including text-only coding models, with streaming and per-request settings
• Tool calling -- session-scoped and request-scoped tool definitions, plus typed tool call/result items
• Embeddings -- single and batch std::vector<float> helpers
• Audio -- offline and real-time streaming audio transcription
• Ranking and typed decisions -- native RankingSession and DecisionSession APIs for CLM and KEV packages
• Execution providers -- discover, download, and register hardware acceleration on demand
• Embedded web service -- optionally expose an OpenAI-compatible local endpoint
• C API -- a stable ABI for other language bindings and low-level integrations

Models with task coding use ChatSession. See coding models for capabilities, registration, and compatibility considerations.

Install From NuGet

You do not need to build Foundry Local from source. Restore the published Microsoft.AI.Foundry.Local.Runtime package and its dependencies with the NuGet CLI:

nuget install Microsoft.AI.Foundry.Local.Runtime -OutputDirectory packages

The Runtime package contains the public C/C++ headers and prebuilt native libraries. It is also the native dependency of the Microsoft.AI.Foundry.Local C# SDK package. Install the Runtime package directly for C and C++ to avoid the managed SDK dependencies. Neither C nor C++ applications need a .NET runtime to use the native libraries.

After restoring the packages:

Asset	Location
C API and C++17 wrapper headers	packages/Microsoft.AI.Foundry.Local.Runtime.<version>/build/native/include/
Foundry Local native library	packages/Microsoft.AI.Foundry.Local.Runtime.<version>/runtimes/<rid>/native/
ONNX Runtime and GenAI dependencies	Matching native assets in the restored Microsoft.ML.OnnxRuntime and Microsoft.ML.OnnxRuntimeGenAI.Foundry packages

Choose the RID for your application's operating system and architecture: win-x64, win-arm64, linux-x64, linux-arm64, or osx-arm64. Add the header directory to your compiler's include path. On Windows, link foundry_local.lib; on Linux, link libfoundry_local.so; on macOS, link libfoundry_local.dylib. Copy the matching native runtime files and their dependency libraries beside your executable or into its loader search path. Keep the restored package versions together; do not mix headers and native libraries from different releases.

See Use In Your Project for a CMake example, Quick Start for C++, and C API for C.

Source Build Prerequisites

The following tools are needed only when developing or customizing the native SDK, not when consuming the published NuGet packages.

• CMake 3.20 or newer
• Python 3.10 or newer to run build.py
• A C++20-capable compiler to build the SDK: Visual Studio 2022/2026 on Windows, Clang 16+ or GCC 12+ on Linux, or Xcode Command Line Tools on macOS
• vcpkg, available through VCPKG_ROOT or explicitly supplied with --vcpkg_root

The build resolves its C++ dependencies from the manifest in vcpkg.json, and obtains the pinned ONNX Runtime and ONNX Runtime GenAI dependencies during CMake configuration.

Linux runtime requirements are architecture-specific: glibc 2.28 or newer on x64, and glibc 2.34 or newer on ARM64. The ARM64 minimum also applies to CPU-only usage because the pinned GenAI runtime requires it.

Chat completions accept a typed chat_template_kwargs JSON object, such as {"enable_thinking": false}, for model-specific template controls. The pinned stable GenAI package supports this feature; a nightly package is not required. Session-level kwargs are defaults: per-request kwargs override the entire object, and {} explicitly clears an inherited default for that request. For OpenAI JSON requests, the payload takes precedence over request options, followed by session defaults. Omitting kwargs inherits the session default, if any.

CUDA acceleration

On Windows and Linux (x64 and ARM64), Foundry Local downloads and verifies only the CUDA execution provider and ORT GenAI CUDA libraries. It does not distribute the CUDA toolkit or cuDNN runtime libraries. Install compatible CUDA and cuDNN runtimes separately before registering the CUDA execution provider.

On Windows, make the runtime DLL directories available through PATH for ORT's provider loader. On x64, also call AddDllDirectory for the CUDA runtime directory before CUDA registration: the GenAI CUDA bridge directly imports the CUDA runtime, and its explicit DLL search policy includes application-added directories but not PATH.

Applications that manage DLL search directories instead of PATH can call AddDllDirectory for each runtime directory and enable these directories with SetDefaultDllDirectories(LOAD_LIBRARY_SEARCH_DEFAULT_DIRS) before CUDA registration. This process-wide setting replaces the standard PATH-based DLL search; register all required runtime directories when using it. It is not needed solely for the x64 bridge load. Foundry Local leaves runtime dependency loading to ORT and GenAI. On Linux, use the system loader configuration or LD_LIBRARY_PATH.

Clearing a Model Cache

Use Model::RemoveFromCache() to clear a completed, interrupted, or corrupted download. It does not require IsCached() to return true. Unload the model first and stop any active downloads; removal rejects loaded models and conflicting downloads. After successful removal, the model handle reports IsCached() == false and its path is cleared. Removal on the same handle is rejected until an active download has finished its callbacks and published its cached state, including cache-hit downloads. Externally registered models must be unregistered instead; their files are not deleted. Applications sharing a cache must coordinate removal with other instances using the model. The download lock serializes file mutations, but does not protect cache-hit reads or models loaded in another process.

GetPath() returns the inference directory, which may be a revision subdirectory. Do not delete that directory manually: download markers and other revisions can live above it. Removal clears the model's cache root, including payloads, inference metadata, download.tmp, blob resume sidecars, and legacy progress markers. An empty root may remain for download-lock safety; it is not considered cached. Shared catalog metadata and other models are retained.

For scanner-discovered locations outside the standard layout, removal also recognizes a parent containing only numeric revision directories (v2, v3, etc.) and download markers or root-level inference_model.json. Matching root metadata can establish ownership without download markers; corrupt metadata requires a marker. Removal locks the parent before inspecting ownership and retains that lock even if unrelated entries or another model's metadata restrict deletion to the discovered inference directory. It never expands to the cache root or across unrelated parent entries.

Native Export Isolation

The shared library exports only FoundryLocalGetApi and FoundryLocalGetVersionString. ELF builds use a version script and --exclude-libs,ALL to keep bundled dependencies and compiler compatibility archives private. macOS/iOS use an exported-symbols allowlist; Windows retains its explicit .def exports. Static and OBJECT libraries remain available to white-box tests. Linux's statically linked internal test executable also localizes archive-owned symbols so telemetry-thread vtables cannot interpose with GenAI. The public export allowlist applies to foundry_local, not separately shipped shared dependencies.

Model-free exact-export checks run in Linux and macOS builds; Linux also checks strong/weak archive isolation, private transport symbols in internal tests, and runtime calls through the public fixture API. ELF checks inspect external definitions only: local symbols retained in .dynsym by GCC compatibility objects are not public exports.

Native non-generative packages

When the web service is enabled, POST /catalogs/local/v1/rank and POST /catalogs/local/v1/systemone resolve the request model as a local-catalog ID or alias. Multi-component packages use the metadata described in docs/non-generative-packages.md. Runtime caches are isolated by canonical model identity and provider; active requests hold lifecycle ownership and block unload. Each loaded typed runtime executes session construction and requests on one dedicated thread so provider and session ownership stay on the same execution thread. Requests resolve a registered local-catalog model ID or alias:

curl -s http://127.0.0.1:5272/catalogs/local/v1/rank \
  -H 'Content-Type: application/json' \
  -d '{"model":"clm","context":{"text":"sample"},"question":"Select","answers":["first","second"],"temperature":1}'

curl -s http://127.0.0.1:5272/catalogs/local/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{"model":"kev","state":"sample","questions":{"q":{"type":"noul","instructions":"Is this valid?"}}}'

Missing fallback configuration is reported as HTTP 503. Invalid contracts, tasks, criteria, or token bounds are HTTP 400. This bounded integration does not add remote catalog submission. Until the corresponding ORT GenAI package is published, configure this branch with --ort_genai_home /path/to/onnxruntime-genai so the native OgaConfig-based NonGenerativeSession API is available.

Cached catalog models can also be used directly without starting the web service. The normal lifecycle uses the same model-owned manager as other tasks: load the model, create a task session, process requests, destroy the session, and unload the model. Session construction requires the selected artifact to be loaded and only acquires a lease on that model-owned runtime.

clm_model.Load();
{
  foundry_local::RankingSession ranking(clm_model);
  const auto ranked = ranking.RankJson(R"({
    "context":{"weather":"heavy rain"},
    "question":"Which activity is more suitable?",
    "answers":["Have a picnic outdoors","Visit an indoor museum"]
  })");
}
clm_model.Unload();

The session already selects the model. The JSON model field may be omitted; a matching bound model ID or alias is accepted, while a different model is rejected rather than routed elsewhere. Temperature precedence is JSON value, then request options, then session defaults. Ordinary text items containing JSON are accepted; responses retain the compatibility OpenAI-JSON text tag. These are Foundry task payloads, not standard OpenAI request schemas.

The SDK methods use the same payload and response shapes as POST /catalogs/local/v1/rank and POST /catalogs/local/v1/systemone. The C ABI remains unchanged: all language bindings use the existing flSession, flRequest, flResponse, and JSON text-item contract.

Build From Source

Building from source is optional. For application development, use the published NuGet packages instead.

From this directory:

# Windows
./build.bat --configure --build --test --config RelWithDebInfo

# Or use the cross-platform Python driver directly.
python build.py --configure --build --test --config RelWithDebInfo

On Linux or macOS:

./build.sh --configure --build --test --config RelWithDebInfo

With no phase flags, build.py runs configure, build, and test. Useful development invocations include:

# Rebuild an already configured tree without running tests.
python build.py --build --config Debug

# Build without examples or the embedded web service.
python build.py --configure --build --skip_examples --skip_service

# Supply local ORT and ORT GenAI source/build roots.
python build.py --configure --build --ort_home C:\src\onnxruntime --ort_genai_home C:\src\onnxruntime-genai

# Generate Windows ARM64 build files.
python build.py --configure --build --arm64

Build output is rooted at build/<platform>/<configuration>/bin/. On Windows, for example, the default library and examples are under build/Windows/RelWithDebInfo/bin/RelWithDebInfo/.

The build places foundry_local alongside its ONNX Runtime and ONNX Runtime GenAI dependencies. Keep these files together when copying a build to another location.

Dependency regression checks

Native Windows x64, macOS ARM64, and Linux CI run the telemetry shutdown and installed nanoarrow consumer regressions. Telemetry tests require TELEMETRY_SHUTDOWN_SDK_SOURCE_DIR to name the pinned telemetry sources with all patches from ports/cpp-client-telemetry/portfile.cmake applied. They link the real Foundry Manager and observe SDK lock, worker, version, and encoder lifetimes. Events traverse normal decoration, queuing, storage, and encoding; only the HTTP transport is replaced with an in-process responder. The upload-after-ci case holds encoding until process exit after a CI-suppressed first Manager.

Windows also runs five DLL lifecycle cases for the Manager owner and standalone telemetry. A mock provider avoids the real SDK workers' DLL pinning so the fixture can exercise actual FreeLibrary teardown. Process-termination cases retain a pending mock uploader callback; they do not access private locks or reproduce lock ownership by terminated threads. Normal destruction and explicit FreeLibrary must each flush, tear down, and release the uploader exactly once. Only Windows process termination abandons cleanup; pending telemetry is not flushed there. The standalone bridge covers teardown inside FreeLibrary; active Manager runtime resources can retain the DLL, so Manager coverage explicitly destroys the instance before unloading.

python tools/run_dependency_tests.py --build build/Windows/RelWithDebInfo --config RelWithDebInfo

The runner rejects missing and skipped cases. Nanoarrow consumers configure, link, and run separately in Debug and Release, with an application-defined flatccrt target declared both before and after package discovery.

Measuring dependency updates

Build baseline and updated SDKs from the same Foundry sources, toolchain, configuration, and ORT/GenAI packages, preserving each native bundle and its vcpkg status file. The sdk-dependency-probe executable loads the actual shared SDK through its public C ABI. Compare the bundles with:

python tools/measure_sdk_dependencies.py `
  --probe build/Windows/RelWithDebInfo/bin/RelWithDebInfo/sdk-dependency-probe.exe `
  --baseline build/baseline-native/foundry_local.dll `
  --candidate build/updated-native/foundry_local.dll `
  --output build/dependency-measurements --samples 10

Add --model-id <exact-catalog-model-id> to time real model downloads into fresh, isolated caches; this is opt-in and may transfer substantial data. Runs alternate baseline/current order and preserve every sample, binary hashes, medians, native payload/ZIP sizes, and (on Windows) unsigned C++ SDK-layout ZIP size including headers, symbols, and the import library. Telemetry is CI-suppressed in both bundles to prevent external collector transmission. These are warm-file-cache, local-host observations, not cold-boot, inference, telemetry-enabled, mobile, or release-signed-package measurements. Network variability is not itself evidence of a dependency performance regression.

Structured coding tools

Structured tool turns on both Engine and Generator use one immutable ToolCallPlan for declaration admission, call-name selection, original-schema validation, and batch acceptance. The model emits a nonempty canonical JSON array inside its published tool markers, for example:

[{"name":"edit","parameters":{"path":"main.py","new_str":"print(\"hello\")\n"}}]

This replaces the model's native XML argument protocol; callers continue using the ordinary Chat Completions, Responses, or SDK tool APIs. Arguments are never coerced, repaired, default-filled, or guessed from XML text.

Generation and validation have different responsibilities. Structural Lark guidance restricts JSON syntax, the canonical envelope, declared/forced tool names, and serial versus parallel calls. It preserves arbitrary argument property order, JSON structural whitespace (bounded formatting runs), and standard string escapes, including printable Unicode escapes. It does not use the runtime's JSON Schema-to-grammar projection: a mask must not turn a valid argument spelling into a different value to satisfy a compiler limitation. The unchanged original parameter schema is validated with jsoncons before any call in the batch is published. Structural guidance alone does not guarantee schema compliance or that the model chose the user's intended argument value.

Automatic tool turns select their response mode explicitly: ordinary assistant text starts with an internal MESSAGE line, while executable calls start with the model's dedicated tool boundary. The runtime removes the text framing line incrementally; callers still receive normal content and tool-call events. Unframed tool-shaped text is rejected instead of silently returned as an answer. Historical assistant text uses the same framing when projected back to the model.

The admitted schema profile supports Draft 2020-12 and supported Draft 7 declarations, local JSON Pointer references, composition, string/numeric/array bounds, and open objects when the original schema permits them. Remote references, recursive references, anchors, custom vocabularies, reserved control keywords, and unsupported content features are rejected explicitly, not silently dropped. Unknown non-reserved keywords and format are treated as annotations; their original values remain in the schema for prompting and validation. Format assertions are not enabled. Schema input is limited to 1 MiB, individual argument JSON to 4 MiB, a canonical batch to 16 MiB, and nesting to 64. Duplicate JSON members are rejected before DOM parsing can erase them.

Only model-published token identities delimit reasoning and executable tool regions. Marker spellings quoted in reasoning or in argument strings remain data, even when the tokenizer marks the published added tokens special:false. Transcript text, tool results, schemas, and historical argument bodies use a separate literal tokenizer; template-owned structure and runtime-owned protocol boundary exemplars use their actual token identities. This separation preserves multiline code, backslashes, Unicode, bracket-prefixed strings, and literal tool/reasoning marker text.

Automatic structured tool turns use whole-turn guidance rather than region guidance, so a marker mentioned in scratchpad text cannot prematurely activate argument decoding. Whole-turn guidance disables speculative drafts; qualify its performance as well as correctness with the intended model. Region-only masks cannot enforce the initial text-versus-tool response mode and are not an automatic tool fallback. Required tools, forced names, and parallel_tool_calls=false remain authoritative.

Concurrent Engine requests wait while resident turns make progress, including background coding-agent compaction. The capacity watchdog bounds stalled capacity, not total queue time; queued requests remain cancelable and do not expire merely because another healthy local generation lasts more than 30 seconds.

An invalid uncommitted call gets at most one guided retry with a bounded schema diagnostic and the remaining output-token budget. Already streamed text/reasoning retains its exact token prefix and is not published twice. No retry is allowed after a tool call was published, cancellation, truncation, stop-sequence interruption, or incompatible explicit/raw guidance. Rejected and interrupted argument payloads never become assistant text. Exhaustion is an explicit error, not an empty successful response.

Recovery also shares implicit classic Generator output limits; it cannot acquire a fresh default allowance or use the retained conversation's full context as an output budget. Engine text turns remain uncapped when the caller leaves their limit unset.

General-chat function qualification is separate from coding-agent qualification. With a locally registered model already loaded, select the ordinary chat cases explicitly:

python sdk_v2/cpp/test/manual/tool_call_gate.py \
  --endpoint http://127.0.0.1:5273/catalogs/local/v1 --model YOUR_MODEL_ID \
  --case chat_function_round_trip --case chat_tool_error --case chat_typed_arguments \
  --case plain_answer_auto --case responses_chat_function_round_trip \
  --output-dir sdk_v2/cpp/build/chat-function-gate

These cases preserve the caller's tool set and automatic selection, correlate real call IDs with results, and check the final answer. Chat Completions runs both streaming modes; the Responses round trip is nonstreaming. Also verify retained typed SDK sessions and fresh processes with the intended runtime packages. Passing these cases does not certify every schema/control combination, a released package, or coding-agent behavior.

Coding-agent qualification requires both protocol fidelity and the real Copilot CLI coding workflow, not just an offline harness self-test. With the server already running:

python sdk_v2\cpp\test\manual\tool_call_gate.py `
  --endpoint http://127.0.0.1:5273/catalogs/local/v1 --model qwen-3.8-27b-dflash2:1 `
  --output-dir sdk_v2\cpp\build\tool-call-gate
python sdk_v2\cpp\test\manual\coding_agent_gate.py `
  --endpoint http://127.0.0.1:5273/catalogs/local/v1 --model qwen-3.8-27b-dflash2:1 `
  --schema-profile native --timeout-seconds 600 --max-output-tokens 8192 `
  --output-dir sdk_v2\cpp\build\coding-agent-gate

Use fresh output directories and run the Copilot gate separately for native, minlength, and oneof. The isolated CLI must read/search files, encounter a real tool error and failing tests, repair the adversarial fixture through real editing tools, and finish with passing tests and exact file bytes. Private HTTP/SSE artifacts must show the intended local provider/model, no provider errors, and no invalid calls. These opt-in gates require the supplied model and installed CLI; they do not download models in CI or certify an unchanged release package. An edit qualifies only with a successful correlated CLI completion and exact full-file original and replacement arguments, including terminal LF; a failed edit followed by shell repair is insufficient. Protocol checks require a successful terminal choice, with [DONE] following it for streaming Chat Completions. The protocol gate also checks optional-property ordering through nonstreaming /responses, requiring a completed response and completed function calls. It checks ordinary function calls with a raw patch descriptor enabled in both streaming and nonstreaming Chat Completions. The failed_tool_recovery_auto case executes a genuinely failing multiply(7,9) callback, retains the identical tools and default automatic selection, and requires a Python final response instead of another tool call. It statically checks the function and expected test without executing model-generated code. Run it alone with --case failed_tool_recovery_auto; a short public-prompt pass does not qualify the real CLI's system prompt, and removing tools on recovery does not qualify either gate. Qualification applies to the model, backend, platform, and exact native binaries recorded for the run. Scripted Generator regressions do not substitute for a trained Generator model gate, and these text-only workflows do not qualify multimodal input. Rebuilt release artifacts must pass the same gates independently.

Quick Start

In-process execution is the default. Call config.SetUseSharedHost(true) before constructing Manager to route the same model/session APIs through the native host shipped beside the runtime. Metadata, discovery and model downloads do not start it; execution, cache removal, EP download/registration or StartWebService() starts or joins a compatible host. Cache removal always runs in the host so it cannot race a newly started host loading the same weights. Compatible applications must use the same Foundry native build and IPC protocol, platform, and effective catalog, data/cache, provider and web configuration. Logs directory/level, effective telemetry settings and additional settings other than UserAgent also separate hosts. Dependency file hashes and packaging differences do not separate clients into different hosts. The first starter selects the host's runtime; its files are verified at launch, and that runtime serves all attached clients. Joining a host does not replace its runtime with the joining client's local dependencies.

The host inherits the first starter's environment, working directory and application identity; later clients do not reconfigure them. Environment-derived settings resolved into the compatibility profile, including telemetry, still affect sharing. Linux shared hosting requires kernel 5.3+ with pidfd_open permitted by the container/seccomp policy, and GNU build IDs in the Foundry, ORT and GenAI runtime modules.

Destroying or shutting down a hosted manager releases only that client. The host drains admitted work and exits after the final owner and a five-second reconnect grace. Keep a manager alive while using its HTTP endpoint: HTTP connections are not owners. Explicit model unload remains shared and rejects in-use models; web-service Stop rejects other owners or active HTTP work. Hosted mode never silently falls back to in-process execution.

The host admits up to 64 connected clients and 64 concurrent IPC operations across clients. Excess operations return an error without replay; callers may retry after work completes. Each connection allows 128 outstanding calls and 256 MiB per encoded message and across pending initial requests. Exceeding an inbound transport limit closes that connection, not the host. Large live inputs should be split into streaming messages. These are admission and wire limits, not quotas on loaded models, decoded items or session history. Session and preflight disposal bypass the execution-worker limit and use a serialized reserved call slot with small, independently bounded requests, so saturation cannot silently orphan a client's native resources.

Model selection checks the model ID, catalog metadata and canonical path, not the contents of every model file. Keep model files immutable while loaded. To replace weights, unload and register the replacement at a new path; overwriting files in place is not a supported revision mechanism.

An embedded HTTP /shutdown request signals shutdown and returns before service teardown. Its SDK owner must observe IsShutdownRequested() and call Shutdown() or destroy the manager to finish releasing the service. Shared hosts reject this endpoint; their teardown follows SDK ownership instead.

Include the C++ wrapper and link the foundry_local shared library. The example below uses a catalog model, downloading it on first use:

#include <foundry_local/foundry_local_cpp.h>

#include <iostream>

using namespace foundry_local;

int main() {
  try {
    Configuration config("my-app");
    Manager manager(std::move(config));

    auto& catalog = manager.GetCatalog();
    auto model = catalog.GetModel("qwen2.5-0.5b");
    if (!model) {
      std::cerr << "Model was not found in the catalog.\n";
      return 1;
    }

    if (!model->IsCached()) {
      model->Download([](float percent) -> int {
        std::cout << "\rDownloading: " << static_cast<int>(percent) << "%" << std::flush;
        return 0;  // Return nonzero to cancel.
      });
      std::cout << "\n";
    }

    model->Load();
    {
      // The session owns conversation state. Creating a session is cheap;
      // loading the model is the expensive operation.
      ChatSession session(*model);
      Request request{UserMessage("Why is the sky blue?")};
      Response response = session.ProcessRequest(request);

      for (const Item& item : response.GetItems()) {
        if (item.GetType() == FOUNDRY_LOCAL_ITEM_MESSAGE) {
          std::cout << item.GetMessage().GetSimpleText() << "\n";
        }
      }
    }
    model->Unload();
  } catch (const Error& error) {
    std::cerr << "Foundry Local error [" << error.Code() << "]: " << error.what() << "\n";
    return 1;
  }
}

Manager must outlive catalog, model, and metadata views obtained from it. SDK wrapper objects use RAII; requests, responses, sessions, and items release their native handles when they leave scope.

Use In Your Project

The public wrapper is header-only. After installing from NuGet, point CMake at the restored Runtime package and your application's RID, and link your executable to the packaged native library:

cmake_minimum_required(VERSION 3.20)
project(my_foundry_app LANGUAGES CXX)

set(FOUNDRY_LOCAL_RUNTIME_DIR "" CACHE PATH "Restored Microsoft.AI.Foundry.Local.Runtime.<version> directory")
set(FOUNDRY_LOCAL_RID "linux-x64" CACHE STRING "Target NuGet runtime identifier")
find_library(FOUNDRY_LOCAL_LIBRARY
    NAMES foundry_local
    PATHS "${FOUNDRY_LOCAL_RUNTIME_DIR}/runtimes/${FOUNDRY_LOCAL_RID}/native"
    NO_DEFAULT_PATH REQUIRED)

add_executable(my_foundry_app main.cc)
target_compile_features(my_foundry_app PRIVATE cxx_std_17)
target_include_directories(my_foundry_app PRIVATE "${FOUNDRY_LOCAL_RUNTIME_DIR}/build/native/include")
target_link_libraries(my_foundry_app PRIVATE "${FOUNDRY_LOCAL_LIBRARY}")

On Windows, place foundry_local.dll, onnxruntime.dll, onnxruntime-genai.dll, and the other native runtime dependencies from the restored packages beside your executable. On Linux and macOS, place libfoundry_local and its neighboring runtime libraries beside the executable or in a loader search path. The shared library uses $ORIGIN on Linux and @loader_path on macOS to locate co-located dependencies.

C API

For a C application, a lower-level integration, or another language binding, include foundry_local/foundry_local_c.h and link the same foundry_local shared library. Use the NuGet installation instructions above to restore the prebuilt native library, matching headers, and runtime dependencies.

Call FoundryLocalGetApi(FOUNDRY_LOCAL_API_VERSION) to obtain the root flApi function table. Check for a null pointer, which indicates that the library does not support the requested API version. The root table exposes manager operations and accessors such as GetConfigurationApi(), GetCatalogApi(), GetModelApi(), GetInferenceApi(), and GetItemApi() for configuration, catalog access, model management, and inference.

The C API uses opaque handles and explicit ownership rather than C++ classes:

• Check the returned flStatus* from status-returning calls. A null status means success; otherwise inspect Status_GetErrorCode() and Status_GetErrorMessage(), then call Status_Release().
• Release owned handles with their corresponding *_Release function. Borrowed pointers and views remain valid only for the lifetime documented in the header.
• Follow the lifetime and callback ownership contracts documented on each function in the public C header. The C++ wrapper automates handle cleanup with RAII.

The C API exposes chat, tool calling, embeddings, audio transcription, and the same model and execution-provider lifecycle used by the C++ wrapper.

Common Workflows

Inspect and manage models

auto& catalog = manager.GetCatalog();

for (const auto& model : catalog.GetModels()) {
  ModelInfo info = model->GetInfo();
  std::cout << info.Alias() << " (" << info.Id() << ")\n";
}

auto model = catalog.GetModel("phi-3.5-mini");
if (model && !model->IsCached()) {
  model->Download();
}
if (model) {
  model->Load();
  // Use a session here.
  model->Unload();
}

Use GetVariants() and SelectVariant() to choose a particular hardware or quantization variant. GetCachedModels() and GetLoadedModels() avoid scanning the full catalog when only local state matters.

Stream chat results

SetStreamingCallback receives items through the native item queue. Return zero to continue or a nonzero value to cancel generation.

ChatSession session(*model);
session.SetStreamingCallback([](flStreamingCallbackData event) -> int {
  flItem* raw_item = nullptr;
  if (detail::item_api()->ItemQueue_TryPop(event.item_queue, &raw_item)) {
    Item item(*raw_item);
    if (item.GetType() == FOUNDRY_LOCAL_ITEM_TEXT) {
      std::cout << item.GetText().text << std::flush;
    }
  }
  return 0;
});

Response response = session.ProcessRequest(Request{UserMessage("Write a short poem.")});

The final Response remains available after streaming completes, including finish reason and token usage.

Configure requests and tools

Apply generation settings to an individual request or to every request on a session:

ChatSession session(*model);
session.AddToolDefinition(ToolDefinition{
    "get_weather",
    "Get the current weather for a city.",
    R"({"type":"object","properties":{"city":{"type":"string"}},"required":["city"]})",
});

Request request{UserMessage("What is the weather in Seattle?")};
RequestOptions options;
options.search.temperature = 0.2f;
options.search.max_output_tokens = 256;
options.tool_choice = FOUNDRY_LOCAL_TOOL_CHOICE_AUTO;
request.SetOptions(options);
Response response = session.ProcessRequest(request);

Read tool calls using Item::GetToolCall(), execute them in application code, and send results on the next turn with Item::ToolResult(call_id, result).

Generate embeddings

EmbeddingsSession session(*model);
std::vector<float> vector = session.Embed("The quick brown fox");
std::vector<std::vector<float>> batch = session.Embed({"First input", "Second input"});

EmbeddingsSession::Embed returns L2-normalized vectors. The model must be an embeddings model and loaded before creating the session.

Manage execution providers

Catalog access does not trigger execution-provider downloads. Discover and register them explicitly when hardware acceleration is needed:

for (const EpInfo& ep : manager.GetDiscoverableEps()) {
  std::cout << ep.name << ": " << (ep.is_registered ? "registered" : "available") << "\n";
}

manager.DownloadAndRegisterEps({}, [](std::string_view name, float percent) {
  std::cout << "\r" << name << ": " << percent << "%" << std::flush;
  return true;  // Return false to cancel.
});

Start the embedded web service

Configuration config("my-app");
config.AddWebServiceEndpoint("http://127.0.0.1:5000");
Manager manager(std::move(config));

manager.StartWebService();
for (const std::string& endpoint : manager.GetWebServiceEndpoints()) {
  std::cout << "Listening on " << endpoint << "\n";
}

manager.StopWebService();

Model management accepts exact versioned model IDs as well as legacy aliases: GET /models/load/{name} and GET /models/unload/{name} use the public catalog; GET /catalogs/local/models/load/{name} and GET /catalogs/local/models/unload/{name} use the local catalog. Encode the entire name as one URL path segment (for example, publisher/model:2 becomes publisher%2Fmodel%3A2). The service percent-decodes once, preserves literal +, and resolves an exact ID before falling back to an alias. Loading an exact variant does not change its alias group's selected variant. Unload still refuses models held by live native sessions. GET /models/loaded and GET /catalogs/local/models/loaded list loaded variant IDs from their respective catalogs, including loaded non-selected variants. A matching ID at a different artifact path is not considered loaded. Adding ?details=true to either loaded-model route instead returns an array of authoritative {"model_id": "...", "model_path": "...", "context_length": 1024} states, qualified by the selected catalog and artifact path. It only observes loaded instances: it does not load weights, create sessions, or fetch metadata. Unknown context limits are omitted. Without details=true, the legacy array of ID strings is unchanged. GET /status advertises "artifact_qualified_model_management": true when these native path-qualified management semantics are available. HTTP model-management clients should check this capability before relying on path-qualified readiness or management requests against an older native library.

Load and unload requests optionally accept the expected_model_path query parameter. Encode the filesystem path as a query value (spaces may be %20 or +; encode literal +, %, &, and #). The service decodes exactly once, canonicalizes the expected and host-resolved cached/BYOM paths, and compares them using native model-loader identity rules before loading, reusing, or unloading weights. Weak canonicalization permits unloading after artifact files or their directory have been removed. A mismatch returns HTTP 409 without changing an existing load; a malformed value returns HTTP 400. Callers omitting this parameter retain the legacy behavior. Shared-serving clients should always supply their selected native Model::GetPath() / SDK Model::path() value: another metadata client can unregister/re-register an ID while a running host still holds its old catalog snapshot. This is a path precondition, not detection of weight-file changes at the same path.

Successful load responses retain status (loaded or already_loaded) and also return the resolved model_id and authoritative filesystem model_path from the loaded instance. Clients should verify both ID and path before binding to that host; registration metadata URI is not an artifact-path contract. For Generator backends, context_length reports the configured total input-plus-output token window used by inference (model.context_length, falling back to legacy search.max_length). Generator explicitly overrides native max_length per request, so a lower artifact search.max_length is not an additional ceiling. This is read from the loaded instance, qualified by model ID and artifact path, without creating a session. It is not a catalog maximum or an independent input/output cap: clients must reserve their requested output tokens within that window. For Engine backends, context_length comes from the existing loaded Engine's native MaxRequestLength capability, also used by session preflight. Observing this cached value creates no request or session. Unknown limits are omitted rather than guessed; the response does not negotiate profiles or verify changes to files at the same artifact path.

POST /shutdown returns {"status":"shutting_down"}, rejects new loads, and cancels tracked sessions. The HTTP worker only requests shutdown: a standalone host must observe Manager::IsShutdownRequested() and release the manager on its owner thread, which stops the service, drains sessions, and unloads models.

Streaming chat completions, Responses, and audio transcriptions cancel their active inference when the SSE client disconnects. The service sends SSE keepalive comments during token silence so a dropped connection can be detected without waiting for the next token. Undrained streams are limited to 1 MiB of buffered events; exceeding the limit cancels inference and replaces pending events with a terminal SSE error. A disconnected stream cannot be resumed. Non-streaming chat completions cancel inference on TCP resets or other transport errors without consuming pending request bytes. A graceful FIN alone does not cancel inference: clients can half-close their sending direction and still read a response, so a graceful full close cannot be distinguished until the response is written. Other non-streaming routes are unchanged. Engine cancellation is cooperative between inference steps; an in-progress prefill step cannot be interrupted before it returns.

Configure with --skip_service when the web service is not required; this removes its oat++ dependency from the build.

Examples

The build produces these example executables in the build output directory:

Example	Demonstrates
basic_chat_example	Catalog lookup, model lifecycle, non-streaming and streaming chat
tool_calling_example	Tool definitions, tool-result turns, and streaming tool calls
embeddings_example	Single and batch embeddings with cosine similarity
realtime_audio_example <audio_file_path>	PCM streaming input and streaming transcription output
kev_decision_example <package_path>	Run a typed KEV decision package through Foundry Local Core

The source for each is under examples/. basic_chat_example honors FOUNDRY_LOCAL_SAMPLE_CACHE_DIR to use an existing model-cache location.

Testing

build.py --test invokes CTest with a 600-second timeout and prints failures. The build creates three test executables:

• foundry_local_tests: implementation-level unit tests
• sdk_integration_tests: public C++ wrapper tests compiled as C++17
• cache_only_tests: cache-only, live-catalog, BYOM, and manager web-service tests

To run one executable directly after a build on Windows:

./build/Windows/RelWithDebInfo/bin/RelWithDebInfo/sdk_integration_tests.exe --gtest_filter="*Chat*"

Some integration tests require cached test models. Set FOUNDRY_TEST_DATA_DIR to the shared test-model cache directory before running them. Tests that cannot use the cache are skipped rather than downloading multi-gigabyte assets.

Project Layout

sdk_v2/cpp/
├── include/foundry_local/  Public C and header-only C++ APIs
├── src/                    Engine implementation
├── examples/               Runnable chat, tool, embedding, and audio programs
├── test/                   Internal and public-SDK GoogleTest suites
├── cmake/                  Shared CMake configuration
├── build.py                Configure, build, test, and cross-compilation driver
├── vcpkg.json              Dependency manifest
└── docs/                   Design notes and implementation documentation

Further Reading

• Wrapper interface design
• Runtime loading behavior
• Item data ownership
• C++ port guide

Non-generative ranking and decisions

The CLM ranking and KEV decision routes translate their existing JSON contracts to the public ORT GenAI NonGenerativeSession JSON boundary. Rendering, tokenization, tensor preparation, pooling, scoring, and typed result construction are owned by ORT GenAI; Foundry only performs contract translation. Runtime provider selection and provider options come from each package's genai_config.json. Package runtimes are cached per canonical catalog identity and evicted by model unload; bounded model-specific caches remain private to ORT GenAI. The HTTP request and response schemas are unchanged.

License

Licensed under the MIT License. See LICENSE.txt.