Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/doc-tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,9 @@ jobs:
DEBUG_MODE: true
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
PYTHONUNBUFFERED: '1'
# Used by the install-agentgateway-binary snippet's `gh run download` of the
# nightly release-binary-linux artifact from agentgateway/agentgateway.
GH_TOKEN: ${{ github.token }}
run: |
FAILED=0
TEST_INDEX=0
Expand Down
3 changes: 3 additions & 0 deletions assets/agw-docs/snippets/config-styles-note.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{{< callout type="info" >}}
Agentgateway supports more than one configuration style. Where a feature can also be configured in the simplified `llm` or `mcp` modes, the examples on this page show each option in tabs. For more information, see [Routing-based configuration]({{< link-hextra path="/llm/configuration-modes/" >}}).
{{< /callout >}}
21 changes: 21 additions & 0 deletions assets/agw-docs/snippets/install-agentgateway-binary.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{{< version include-if="main" >}}
# Install the agentgateway binary from the latest main (nightly) build.
# The nightly build publishes a container image tagged 'latest-dev'; extract the
# binary from that image. The GitHub release assets only exist for tagged
# releases, not for the in-development 'main' version.
mkdir -p "$HOME/.local/bin"
export PATH="$HOME/.local/bin:$PATH"
docker rm -f agw-extract >/dev/null 2>&1 || true
docker create --name agw-extract cr.agentgateway.dev/agentgateway:latest-dev
docker cp agw-extract:/app/agentgateway "$HOME/.local/bin/agentgateway"
docker rm agw-extract
chmod +x "$HOME/.local/bin/agentgateway"
{{< /version >}}
{{< version exclude-if="main" >}}
# Install the latest released agentgateway binary to local bin without sudo.
mkdir -p "$HOME/.local/bin"
export PATH="$HOME/.local/bin:$PATH"
BINARY_URL="https://github.com/agentgateway/agentgateway/releases/latest/download/agentgateway-$(uname -s | tr '[:upper:]' '[:lower:]')-$(uname -m | sed 's/x86_64/amd64/')"
curl -sL "$BINARY_URL" -o "$HOME/.local/bin/agentgateway"
chmod +x "$HOME/.local/bin/agentgateway"
{{< /version >}}
279 changes: 251 additions & 28 deletions content/docs/standalone/main/configuration/backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,25 @@ title: Backends
weight: 13
description: Configure backends to route traffic to hostnames, LLM providers, and MCP servers.
prev: /configuration/listeners
test:
backends:
- file: content/docs/standalone/main/configuration/backends.md
path: backends
---

Agentgateway {{< gloss "Backend" >}}backends{{< /gloss >}} control where traffic is routed to.
Agentgateway supports a variety of backends, such as simple hostnames and IP addresses, {{< gloss "Provider" >}}LLM providers{{< /gloss >}}, and MCP servers.

{{< reuse "agw-docs/snippets/config-styles-note.md" >}}

{{< doc-test paths="backends" >}}
{{< reuse "agw-docs/snippets/install-agentgateway-binary.md" >}}
export OPENAI_API_KEY="${OPENAI_API_KEY:-dummy}"
{{< /doc-test >}}

## Static Hosts

The simplest form of backend is a static hostname or IP address. For example:
The simplest form of backend is a static hostname or IP address. Static hosts are a routing-based backend, so they are configured under `binds`; the simplified `llm` and `mcp` modes model only LLM providers and MCP targets. For example:

```yaml
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
Expand All @@ -26,53 +37,265 @@ binds:
weight: 9
```

{{< doc-test paths="backends" >}}
# WHAT THIS TEST VALIDATES:
# * The static host backend example config is accepted by agentgateway.
# WHAT THIS TEST DOES NOT VALIDATE (and why):
# * That traffic is actually routed/weighted to the hosts at runtime — requires
# reachable backends the page omits.
cat <<'EOF' > config.yaml
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
binds:
- port: 3000
listeners:
- protocol: HTTP
routes:
- backends:
- host: example.com:8080
weight: 1
- host: 127.0.0.1:80
weight: 9
EOF
agentgateway -f config.yaml --validate-only
{{< /doc-test >}}

## MCP Servers

The MCP backend allows you to connect to an MCP server.
Below shows a simple example, exposing a local and remote MCP server.
See the [MCP connectivity guide]({{< link-hextra path="/mcp/" >}}) for more information.

{{< tabs >}}
{{< tab name="Simplified (MCP)" >}}
```yaml
backends:
- mcp:
targets:
- name: stdio-server
stdio:
cmd: npx
args: ["@modelcontextprotocol/server-everything"]
- name: http-server
mcp:
host: https://example.com/mcp
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
mcp:
port: 3000
targets:
- name: stdio-server
stdio:
cmd: npx
args: ["@modelcontextprotocol/server-everything"]
- name: http-server
mcp:
host: https://example.com/mcp
```
{{< /tab >}}
{{< tab name="Routing-based" >}}
```yaml
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
binds:
- port: 3000
listeners:
- routes:
- backends:
- mcp:
targets:
- name: stdio-server
stdio:
cmd: npx
args: ["@modelcontextprotocol/server-everything"]
- name: http-server
mcp:
host: https://example.com/mcp
```
{{< /tab >}}
{{< /tabs >}}

{{< doc-test paths="backends" >}}
# WHAT THIS TEST VALIDATES:
# * The MCP backend example (stdio + remote MCP targets) is accepted by
# agentgateway in both the routing-based (binds) and simplified MCP (mcp) forms.
# WHAT THIS TEST DOES NOT VALIDATE (and why):
# * That the MCP targets actually start/connect at runtime — requires the npx
# command and remote server the page does not stand up.
cat <<'EOF' > config2.yaml
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
binds:
- port: 3000
listeners:
- routes:
- backends:
- mcp:
targets:
- name: stdio-server
stdio:
cmd: npx
args: ["@modelcontextprotocol/server-everything"]
- name: http-server
mcp:
host: https://example.com/mcp
EOF
agentgateway -f config2.yaml --validate-only

cat <<'EOF' > config2-simplified.yaml
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
mcp:
port: 3000
targets:
- name: stdio-server
stdio:
cmd: npx
args: ["@modelcontextprotocol/server-everything"]
- name: http-server
mcp:
host: https://example.com/mcp
EOF
agentgateway -f config2-simplified.yaml --validate-only
{{< /doc-test >}}

### Session routing

By default, MCP backends use stateful session routing, where the gateway tracks session IDs and routes subsequent requests to the same upstream. For upstreams that do not maintain server-side session state, you can set `statefulMode: Stateless`. In stateless mode, the gateway automatically wraps each request with an initialization sequence, so the upstream server processes every request independently.
By default, MCP backends use stateful session routing, where the gateway tracks session IDs and routes subsequent requests to the same upstream. For upstreams that do not maintain server-side session state, you can set `statefulMode: stateless`. In stateless mode, the gateway automatically wraps each request with an initialization sequence, so the upstream server processes every request independently.

{{< tabs >}}
{{< tab name="Simplified (MCP)" >}}
```yaml
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
mcp:
port: 3000
statefulMode: stateless
targets:
- name: openapi-server
openapi:
host: petstore3.swagger.io:443
schema:
url: https://petstore3.swagger.io/api/v3/openapi.json
```
{{< /tab >}}
{{< tab name="Routing-based" >}}
```yaml
backends:
- mcp:
statefulMode: Stateless
targets:
- name: openapi-server
openapi:
schema:
url: https://petstore3.swagger.io/api/v3/openapi.json
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
binds:
- port: 3000
listeners:
- routes:
- backends:
- mcp:
statefulMode: stateless
targets:
- name: openapi-server
openapi:
host: petstore3.swagger.io:443
schema:
url: https://petstore3.swagger.io/api/v3/openapi.json
```
{{< /tab >}}
{{< /tabs >}}

{{< doc-test paths="backends" >}}
# WHAT THIS TEST VALIDATES:
# * The stateless session-routing MCP backend example is accepted by agentgateway
# in both the routing-based (binds) and simplified MCP (mcp) forms.
# WHAT THIS TEST DOES NOT VALIDATE (and why):
# * That stateless wrapping actually occurs at runtime — requires the OpenAPI
# upstream and live MCP traffic the page omits.
cat <<'EOF' > config3.yaml
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
binds:
- port: 3000
listeners:
- routes:
- backends:
- mcp:
statefulMode: stateless
targets:
- name: openapi-server
openapi:
host: petstore3.swagger.io:443
schema:
url: https://petstore3.swagger.io/api/v3/openapi.json
EOF
agentgateway -f config3.yaml --validate-only

cat <<'EOF' > config3-simplified.yaml
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
mcp:
port: 3000
statefulMode: stateless
targets:
- name: openapi-server
openapi:
host: petstore3.swagger.io:443
schema:
url: https://petstore3.swagger.io/api/v3/openapi.json
EOF
agentgateway -f config3-simplified.yaml --validate-only
{{< /doc-test >}}

## LLM Providers

Agentgateway natively supports connecting to LLM providers, such as OpenAI and Anthropic.
Below shows a simple example, connecting to OpenAI.
See the [LLM consumption guide]({{< link-hextra path="/llm/" >}}) for more information.

{{< tabs >}}
{{< tab name="Simplified (LLM)" >}}
```yaml
backends:
- ai:
provider:
openAI:
model: gpt-3.5-turbo
policies:
backendAuth:
key: "$OPENAI_API_KEY"
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
llm:
models:
- name: openai
provider: openAI
params:
model: gpt-3.5-turbo
apiKey: "$OPENAI_API_KEY"
```
{{< /tab >}}
{{< tab name="Routing-based" >}}
```yaml
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
binds:
- port: 3000
listeners:
- routes:
- backends:
- ai:
name: openai
provider:
openAI:
model: gpt-3.5-turbo
policies:
backendAuth:
key: "$OPENAI_API_KEY"
```
{{< /tab >}}
{{< /tabs >}}

{{< doc-test paths="backends" >}}
# WHAT THIS TEST VALIDATES:
# * The OpenAI LLM provider example is accepted by agentgateway in both the
# routing-based (ai backend) and simplified LLM (llm.models) forms.
# WHAT THIS TEST DOES NOT VALIDATE (and why):
# * That requests are actually proxied to OpenAI at runtime — requires a real
# OPENAI_API_KEY and live LLM traffic the page omits.
cat <<'EOF' > config4.yaml
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
binds:
- port: 3000
listeners:
- routes:
- backends:
- ai:
name: openai
provider:
openAI:
model: gpt-3.5-turbo
policies:
backendAuth:
key: "$OPENAI_API_KEY"
EOF
agentgateway -f config4.yaml --validate-only

cat <<'EOF' > config4-simplified.yaml
# yaml-language-server: $schema=https://agentgateway.dev/schema/config
llm:
models:
- name: openai
provider: openAI
params:
model: gpt-3.5-turbo
apiKey: "$OPENAI_API_KEY"
EOF
agentgateway -f config4-simplified.yaml --validate-only
{{< /doc-test >}}
Loading
Loading