Skip to content

About

for Azure SRE Agent and Azure Copilot observerbility Agent demo repo

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

AzureSREAgentDemoLab - SRE Training Microservices

日本語版は README_ja.md を参照してください。

A microservices application used for validating and teaching the Azure SRE Agent and the Azure Copilot Observability Agent.

Architecture

graph TB
    Client[Client / Browser] --> AGC[Application Gateway<br/>for Containers]
    
    subgraph AKS[Azure Kubernetes Service]
        ALB[ALB Controller] -.->|config sync| AGC
        AGC --> CS[Catalog Service<br/>:5001]
        AGC --> OS[Order Service<br/>:5002]
        AGC --> NS[Notification Service<br/>:5003]
    end
    
    CS --> CosmosDB[(Azure Cosmos DB)]
    OS --> CosmosDB
    NS --> CosmosDB
    
    CS --> OTEL[OpenTelemetry Collector]
    OS --> OTEL
    NS --> OTEL
    
    subgraph Auth[Microsoft Entra ID]
        EntraID[JWT Bearer Auth]
    end
    
    CS -.-> EntraID
    OS -.-> EntraID
    NS -.-> EntraID
Loading

Note: The community edition of Ingress NGINX was retired in March 2026. This project uses Application Gateway for Containers (AGC) together with the Gateway API.

Services

Service Port Endpoints Description
CatalogService 5001 10 Product catalog, categories, inventory management
OrderService 5002 8 Order management, status tracking
NotificationService 5003 6 Notification delivery, read-state management

Technology stack

  • Runtime: .NET 10
  • Framework: ASP.NET Core Minimal APIs
  • Authentication: Microsoft Entra ID (JWT bearer authentication)
  • Data store: Azure Cosmos DB (NoSQL API)
  • Telemetry: OpenTelemetry (traces + metrics + logs)
  • Logging: Serilog (structured logging)
  • Containers: Docker (multi-stage builds)
  • Orchestration: Kubernetes (AKS)

API endpoints

CatalogService (/api/products, /api/categories, /api/inventory)

Method Path Description
GET /api/products List products
GET /api/products/{id} Get product details
POST /api/products Create a product
PUT /api/products/{id} Update a product
DELETE /api/products/{id} Delete a product
GET /api/products/search?q={query} Search products ⚠️
GET /api/categories List categories
GET /api/categories/{id} Get category details
GET /api/inventory/{productId} Check inventory
PUT /api/inventory/{productId} Update inventory

OrderService (/api/orders)

Method Path Description
GET /api/orders List orders
GET /api/orders/{id} Get order details
POST /api/orders Create an order
PUT /api/orders/{id}/status Update order status
DELETE /api/orders/{id} Delete an order
GET /api/orders/user/{userId} List orders by user
POST /api/orders/{id}/cancel Cancel an order
GET /api/orders/{id}/total Calculate order total

NotificationService (/api/notifications)

Method Path Description
GET /api/notifications List notifications
GET /api/notifications/{id} Get notification details
POST /api/notifications Create a notification
PUT /api/notifications/{id}/read Mark as read
GET /api/notifications/user/{userId} List notifications by user
DELETE /api/notifications/{id} Delete a notification

Health check endpoints

Each service exposes a consistent set of health-check endpoints that correspond to the Kubernetes liveness and readiness probes.

GET /health — Liveness probe

Checks only whether the process is alive. It does not check dependencies.

{
  "status": "Healthy",
  "checks": []
}

GET /health/ready — Readiness probe

Returns Healthy only when all of the following conditions are met.

Check Description
cosmosdb CosmosClient.ReadAccountAsync() succeeds
startup Cosmos DB initialization and seeding are complete
{
  "status": "Healthy",
  "checks": [
    { "name": "cosmosdb", "status": "Healthy", "description": "Cosmos DB is reachable.", "duration": 42.3 },
    { "name": "startup", "status": "Healthy", "description": "Startup initialization complete.", "duration": 0.1 }
  ]
}

Local development

Prerequisites (before deploying to Azure)

  • .NET 10 SDK
  • Docker Desktop
  • (Optional) Azure Cosmos DB Emulator

Start with docker-compose

docker compose up --build

The services are accessible on the following ports:

In development mode, authentication is skipped (Authentication:DisableAuth=true).

Note: In the docker-compose environment, CosmosDb:AllowInsecureCertificate=true is set. This allows the self-signed certificate of the Linux Cosmos DB emulator. The connection mode (Gateway) can be configured independently with CosmosDb:ConnectionMode (if unset and AllowInsecureCertificate=true, Gateway is the default; otherwise Direct is the default). Never use this in production.

Run individual services

dotnet run --project src/CatalogService
dotnet run --project src/OrderService
dotnet run --project src/NotificationService

Build

dotnet build AzureSREAgentDemoLab.slnx

Deploy with Azure Developer CLI + Bicep

This repository includes an Azure Developer CLI (azd) + Bicep configuration that lets you deploy repeatedly to multiple environments for training purposes.

Azure resources created

  • Azure Container Apps Environment
  • Azure Container Apps x 3
    • Catalog
    • Order
    • Notification
  • Azure Container Registry (Basic)
  • Azure Cosmos DB for NoSQL (EnableServerless)
  • Cosmos SQL Database
  • Log Analytics Workspace
  • Application Insights

Design principles

  • Cost optimization first
    • Uses Container Apps instead of AKS
    • Non-prod environments use minReplicas = 0
    • ACR is Basic
    • Cosmos DB is serverless
  • No Private Endpoints
  • Default region is westus3
    • westus3 is used as the IaC default
    • Creating environments with azd env new <env> -l westus3 is recommended
  • Multi-environment support
    • Resource names are separated based on AZURE_ENV_NAME
    • dev / test / prod / workshop-a and similar can run in parallel

Prerequisites

  • Azure Developer CLI (azd)
  • Azure CLI (az)
  • Docker Desktop
  • .NET 10 SDK
  • Access to an Azure subscription

Per-environment setup

azd creates a .azure/<environment-name>/.env file for each environment.

First, create the environment.

azd env new dev -l westus3

Next, set the required values for the environment. scripts/setup-entra-app.ps1 automates the Entra ID registration and the value configuration (see docs/entra-app-setup.md for details).

azd env set ENTRA_TENANT_ID <your-tenant-id>
azd env set ENTRA_CLIENT_ID <your-client-id>
azd env set ENTRA_AUDIENCE <identifier-uri-from-entra-registration>

For ENTRA_AUDIENCE, specify the App ID URI (identifier URI) of the Entra ID app registration. If you used scripts/setup-entra-app.ps1 it has the form api://<your-client-id>; if you used the Bicep template it has the form api://<sanitized-display-name>. The JWT aud validation must match this value.

You can set additional values as needed.

azd env set ENVIRONMENT_TYPE dev
azd env set DISABLE_AUTH false
azd env set OPEN_TELEMETRY_ENDPOINT <optional-otlp-endpoint>

Reference: What azd actually reads is .azure/<environment-name>/.env. Keep a note of the values you need.

Pre-deployment validation

The following have been validated locally:

  • infra/main.bicep: no diagnostic errors
  • infra/modules/container-app.bicep: no diagnostic errors
  • dotnet build AzureSREAgentDemoLab.slnx: succeeds

However, validating against Azure requires an azd environment configured with real Entra ID values.

Deployment flow

  1. Create the environment
  2. Apply the required azd env set values
  3. Sign in to Azure
  4. Run the deployment
az login
azd up

Example: adding more environments

azd env new test -l westus3
azd env new prod -l westus3
azd env new workshop-a -l westus3

Because .azure/<environment-name>/.env is separated per environment, this works well for parallel validation in training scenarios.

Key files

  • azure.yaml
    • azd service definitions
  • infra/main.bicep
    • Shared Azure resources + deployment of each service
  • infra/modules/container-app.bicep
    • Common Container App module for each API service
  • infra/main.parameters.json
    • Passes azd environment variables to Bicep parameters

AKS deployment

1. Provision Azure resources (Bicep)

Provisions AKS, Application Gateway for Containers, VNet, Cosmos DB, ACR, and more in one go.

Bash:

# Create the resource group
az group create --name rg-azure-sre-agent-demo-lab --location westus3

# Deploy the infrastructure with Bicep
az deployment group create \
  --name main-aks \
  --resource-group rg-azure-sre-agent-demo-lab \
  --template-file infra/main-aks.bicep \
  --parameters \
    environmentName=dev \
    entraTenantId=<YOUR_TENANT_ID> \
    entraClientId=<YOUR_CLIENT_ID> \
    entraAudience=<YOUR_AUDIENCE>

PowerShell:

# Create the resource group
az group create --name rg-azure-sre-agent-demo-lab --location westus3

# Deploy the infrastructure with Bicep
az deployment group create `
  --name main-aks `
  --resource-group rg-azure-sre-agent-demo-lab `
  --template-file infra/main-aks.bicep `
  --parameters `
    environmentName=dev `
    entraTenantId=<YOUR_TENANT_ID> `
    entraClientId=<YOUR_CLIENT_ID> `
    entraAudience=<YOUR_AUDIENCE>

After the deployment completes, capture the following output values:

Bash:

# Inspect the deployment outputs
az deployment group show \
  --resource-group rg-azure-sre-agent-demo-lab \
  --name main-aks \
  --query properties.outputs -o json

PowerShell:

# Inspect the deployment outputs
az deployment group show `
  --resource-group rg-azure-sre-agent-demo-lab `
  --name main-aks `
  --query properties.outputs -o json

Key output values:

Output Description
AKS_CLUSTER_NAME AKS cluster name
ACR_LOGIN_SERVER ACR login server
AGC_RESOURCE_ID Application Gateway for Containers resource ID
AGC_FRONTEND_NAME AGC frontend name referenced by the Gateway
AGC_FRONTEND_FQDN AGC frontend FQDN
ALB_IDENTITY_CLIENT_ID Managed identity for the ALB Controller
COSMOS_ACCOUNT_NAME Cosmos DB account name

2. Entra ID app registration

2-1. Create the app registration

# Create the app registration and capture the application ID
APP_ID=$(az ad app create --display-name "<your-entra-app-name>" --query appId -o tsv)
TENANT_ID=$(az account show --query tenantId -o tsv)

echo "TenantId : $TENANT_ID"
echo "ClientId : $APP_ID"
# Create the app registration and capture the application ID
$APP_ID = (az ad app create --display-name "<your-entra-app-name>" --query appId -o tsv)
$TENANT_ID = (az account show --query tenantId -o tsv)

Write-Host "TenantId : $TENANT_ID"
Write-Host "ClientId : $APP_ID"

2-2. Configure the App ID URI and scope

Set the App ID URI to protect it as an API.

Bash:

# Set the App ID URI (api://<appId> form)
az ad app update --id $APP_ID --identifier-uris "api://${APP_ID}"

PowerShell:

# Set the App ID URI (api://<appId> form)
az ad app update --id $APP_ID --identifier-uris "api://$APP_ID"

If you want to acquire a user token with Azure CLI login credentials, you also need a delegated scope (access_as_user). It is created automatically when you use infra/modules/entra-app.bicep. If you registered the app manually, add it with the following commands.

Bash:

# Get the app registration Object ID (different from appId)
OBJ_ID=$(az ad app show --id $APP_ID --query id -o tsv)

# Read the existing oauth2PermissionScopes and add access_as_user only if it is missing
APP_JSON=$(az ad app show --id $APP_ID -o json)
EXISTING_SCOPE_ID=$(echo "$APP_JSON" | jq -r '.api.oauth2PermissionScopes[]? | select(.value == "access_as_user") | .id' | head -n 1)
if [ -n "$EXISTING_SCOPE_ID" ]; then
  echo "access_as_user scope already exists: $EXISTING_SCOPE_ID"
else
  SCOPE_ID=$(python3 -c "import uuid; print(uuid.uuid4())")
  SCOPES=$(echo "$APP_JSON" | jq --arg scopeId "$SCOPE_ID" '
    (.api.oauth2PermissionScopes // []) + [
      {
        id: $scopeId,
        adminConsentDisplayName: "Access API as user",
        adminConsentDescription: "Allows the app to access the API on behalf of the signed-in user",
        userConsentDisplayName: "Access API as you",
        userConsentDescription: "Allows the app to access the API on your behalf",
        value: "access_as_user",
        type: "User",
        isEnabled: true
      }
    ]')
  BODY=$(jq -n --argjson scopes "$SCOPES" '{api: {oauth2PermissionScopes: $scopes}}')
  az rest --method PATCH \
    --uri "https://graph.microsoft.com/v1.0/applications/$OBJ_ID" \
    --headers "Content-Type=application/json" \
    --body "$BODY"
fi

PowerShell:

# Get the app registration Object ID (different from appId)
$OBJ_ID = (az ad app show --id $APP_ID --query id -o tsv)

# Read the existing oauth2PermissionScopes and add access_as_user only if it is missing
$appJson = az ad app show --id $APP_ID -o json | ConvertFrom-Json
$existingScopes = @($appJson.api.oauth2PermissionScopes)
$existingScope = $existingScopes | Where-Object { $_.value -eq "access_as_user" } | Select-Object -First 1

if ($existingScope) {
  Write-Host "access_as_user scope already exists: $($existingScope.id)"
}
else {
  $scopeId = [System.Guid]::NewGuid().ToString()
  $newScope = [ordered]@{
    id                      = $scopeId
    adminConsentDisplayName = "Access API as user"
    adminConsentDescription = "Allows the app to access the API on behalf of the signed-in user"
    userConsentDisplayName  = "Access API as you"
    userConsentDescription  = "Allows the app to access the API on your behalf"
    value                   = "access_as_user"
    type                    = "User"
    isEnabled               = $true
  }

  $patchBody = @{
    api = @{
      oauth2PermissionScopes = @($existingScopes + $newScope)
    }
  } | ConvertTo-Json -Depth 10

  $tmpScopeFile = [System.IO.Path]::GetTempFileName()
  $patchBody | Out-File -FilePath $tmpScopeFile -Encoding utf8NoBOM

  try {
    az rest --method PATCH `
      --uri "https://graph.microsoft.com/v1.0/applications/$OBJ_ID" `
      --headers "Content-Type=application/json" `
      --body "@$tmpScopeFile"
  }
  finally {
    Remove-Item $tmpScopeFile -ErrorAction SilentlyContinue
  }
}

After adding the scope, perform user consent or admin consent according to your tenant's consent policy.

Next, add an app role for service-to-service (client_credentials) access. In the Azure portal, go to App registrations → App roles → Create app role and enter the following values.

Field Value
Display name Access AzureSREAgentDemoLab API
Allowed member types Applications
Value Api.Access
Description Allows a service to call the AzureSREAgentDemoLab APIs
Do you want to enable this app role? On

Note: You can also add the role via a JSON patch with az ad app update, but the portal operation is the most reliable.

2-3. Create the service principal and client secret

Bash:

# Create the service principal
az ad sp create --id $APP_ID

# Create a client secret (expires in 1 year)
CLIENT_SECRET=$(az ad app credential reset --id $APP_ID --years 1 --query password -o tsv)

echo "Client secret created. Store it in a safe place."
echo "Audience     : api://$APP_ID"

PowerShell:

# Create the service principal
az ad sp create --id $APP_ID

# Create a client secret (expires in 1 year)
$CLIENT_SECRET = (az ad app credential reset --id $APP_ID --years 1 --query password -o tsv)

Write-Host "Client secret created. Store it in a safe place."
Write-Host "Audience     : api://$APP_ID"

⚠️ CLIENT_SECRET is shown only once. Be sure to store it in a safe place.

2-4. Grant admin consent for the app role

Prerequisite: This step uses jq. If it is not installed: sudo apt-get install jq (Debian/Ubuntu) or brew install jq (macOS). An alternative command for when jq is unavailable is shown below.

Bash:

# Get the service principal Object ID
SP_OID=$(az ad sp show --id $APP_ID --query id -o tsv)

# Get the appRoleId of the app role
ROLE_ID=$(az ad app show --id $APP_ID \
  --query "appRoles[?value=='Api.Access'].id" -o tsv)

# Grant the role to your own service principal (admin consent) — using jq
az rest --method POST \
  --uri "https://graph.microsoft.com/v1.0/servicePrincipals/${SP_OID}/appRoleAssignments" \
  --body "$(jq -n \
    --arg pid "$SP_OID" \
    --arg rid "$SP_OID" \
    --arg aid "$ROLE_ID" \
    '{principalId: $pid, resourceId: $rid, appRoleId: $aid}')"

# If jq is unavailable, build the JSON with python3
# az rest --method POST \
#   --uri "https://graph.microsoft.com/v1.0/servicePrincipals/${SP_OID}/appRoleAssignments" \
#   --body "$(python3 -c "
# import json, sys
# print(json.dumps({'principalId': '${SP_OID}', 'resourceId': '${SP_OID}', 'appRoleId': '${ROLE_ID}'}))")"

PowerShell:

# Get the service principal Object ID
$SP_OID = (az ad sp show --id $APP_ID --query id -o tsv)

# Get the appRoleId of the app role
$ROLE_ID = (az ad app show --id $APP_ID --query "appRoles[?value=='Api.Access'].id" -o tsv)

# Grant the role to your own service principal (admin consent)
$body = @{
    principalId = $SP_OID
    resourceId  = $SP_OID
    appRoleId   = $ROLE_ID
} | ConvertTo-Json

$tmpBodyFile = [System.IO.Path]::GetTempFileName()
$body | Out-File -FilePath $tmpBodyFile -Encoding utf8NoBOM

try {
  az rest --method POST `
    --uri "https://graph.microsoft.com/v1.0/servicePrincipals/$SP_OID/appRoleAssignments" `
    --headers "Content-Type=application/json" `
    --body "@$tmpBodyFile"
}
finally {
  Remove-Item $tmpBodyFile -ErrorAction SilentlyContinue
}

2-5. Update the Kubernetes secrets

Reflect the app registration information into the Kubernetes secrets.

⚠️ Prerequisites: Before running this step, install and verify the following.

Prerequisite: install kubectl and helm

Install the tools required to manage the AKS cluster.

Step 2-5-1: Verify the connection to the AKS cluster

Set the kubectl context using the AKS cluster name output by the Bicep deployment.

Bash:

# Get the AKS cluster name
AKS_NAME=$(az deployment group show -g rg-azure-sre-agent-demo-lab -n main-aks --query properties.outputs.akS_CLUSTER_NAME.value -o tsv)

# Get the AKS cluster credentials
az aks get-credentials --resource-group rg-azure-sre-agent-demo-lab --name $AKS_NAME --overwrite-existing

# Verify the connection
kubectl cluster-info
kubectl config current-context

# Create the namespace for the secrets first
kubectl apply -f k8s/namespace.yaml

PowerShell:

# Get the AKS cluster name
$AKS_NAME = (az deployment group show -g rg-azure-sre-agent-demo-lab -n main-aks --query "properties.outputs.akS_CLUSTER_NAME.value" -o tsv)

# Get the AKS cluster credentials
az aks get-credentials --resource-group rg-azure-sre-agent-demo-lab --name $AKS_NAME --overwrite-existing

# Verify the connection
kubectl cluster-info
kubectl config current-context

# Create the namespace for the secrets first
kubectl apply -f k8s/namespace.yaml

If the connection succeeds, the message above is displayed. If it fails, check the resource group name, the cluster name, and the subscription.

Step 2-5-2: Create the Kubernetes secrets

Bash:

kubectl create secret generic entra-id-secret \
  --namespace azure-sre-agent-demo-lab \
  --from-literal=TenantId="${TENANT_ID}" \
  --from-literal=ClientId="${APP_ID}" \
  --from-literal=Audience="api://${APP_ID}" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl create secret generic cosmos-db-secret \
  --namespace azure-sre-agent-demo-lab \
  --from-literal=ConnectionString="$(az cosmosdb keys list \
    --name $(az deployment group show -g rg-azure-sre-agent-demo-lab -n main-aks --query properties.outputs.cosmoS_ACCOUNT_NAME.value -o tsv) \
    --resource-group rg-azure-sre-agent-demo-lab \
    --type connection-strings \
    --query connectionStrings[0].connectionString -o tsv)" \
  --dry-run=client -o yaml | kubectl apply -f -

PowerShell:

# Use the values captured in the PowerShell steps in 2-1
# If you run this in a separate session, set them first
# $APP_ID = "<YOUR_CLIENT_ID>"
# $TENANT_ID = (az account show --query tenantId -o tsv)

# Get the Cosmos DB connection string
$COSMOS_ACCOUNT_NAME = (az deployment group show -g rg-azure-sre-agent-demo-lab -n main-aks --query "properties.outputs.COSMOS_ACCOUNT_NAME.value" -o tsv)
$CONNECTION_STRING = (az cosmosdb keys list `
  --name $COSMOS_ACCOUNT_NAME `
  --resource-group rg-azure-sre-agent-demo-lab `
  --type connection-strings `
  --query "connectionStrings[0].connectionString" -o tsv)

# Create entra-id-secret
kubectl create secret generic entra-id-secret `
  --namespace azure-sre-agent-demo-lab `
  --from-literal="TenantId=$TENANT_ID" `
  --from-literal="ClientId=$APP_ID" `
  --from-literal="Audience=api://$APP_ID" `
  --dry-run=client -o yaml | kubectl apply -f -

# Create cosmos-db-secret
kubectl create secret generic cosmos-db-secret `
  --namespace azure-sre-agent-demo-lab `
  --from-literal="ConnectionString=$CONNECTION_STRING" `
  --dry-run=client -o yaml | kubectl apply -f -

Note: k8s/catalog-service.yaml used to contain a Secret template with placeholders, but that template has been separated into k8s/secrets.template.yaml. After registering the real values with the kubectl commands above, do not apply k8s/secrets.template.yaml (it would overwrite them).

3. Build and push the container images

Bash:

# Bicep deployment name (do not hardcode)
DEPLOYMENT_NAME="<YOUR_BICEP_DEPLOYMENT_NAME>"

# Get the ACR from the deployment outputs
ACR_NAME=$(az deployment group show -g rg-azure-sre-agent-demo-lab -n "$DEPLOYMENT_NAME" --query properties.outputs.ACR_NAME.value -o tsv)
ACR_LOGIN_SERVER=$(az deployment group show -g rg-azure-sre-agent-demo-lab -n "$DEPLOYMENT_NAME" --query properties.outputs.ACR_LOGIN_SERVER.value -o tsv)

# az acr login takes the registry name (xxxx)
az acr login --name "$ACR_NAME"

docker build -t catalog-service:latest -f src/CatalogService/Dockerfile .
docker build -t order-service:latest -f src/OrderService/Dockerfile .
docker build -t notification-service:latest -f src/NotificationService/Dockerfile .

docker tag catalog-service:latest "$ACR_LOGIN_SERVER/catalog-service:latest"
docker tag order-service:latest "$ACR_LOGIN_SERVER/order-service:latest"
docker tag notification-service:latest "$ACR_LOGIN_SERVER/notification-service:latest"

docker push "$ACR_LOGIN_SERVER/catalog-service:latest"
docker push "$ACR_LOGIN_SERVER/order-service:latest"
docker push "$ACR_LOGIN_SERVER/notification-service:latest"

PowerShell:

# Get the ACR from the resource group
$ACR_NAME = (az acr list -g rg-azure-sre-agent-demo-lab --query "[0].name" -o tsv)
$ACR_LOGIN_SERVER = (az acr show -g rg-azure-sre-agent-demo-lab -n $ACR_NAME --query "loginServer" -o tsv)

# az acr login takes the registry name (xxxx)
az acr login --name $ACR_NAME

docker build -t catalog-service:latest -f src/CatalogService/Dockerfile .
docker build -t order-service:latest -f src/OrderService/Dockerfile .
docker build -t notification-service:latest -f src/NotificationService/Dockerfile .

docker tag catalog-service:latest "$ACR_LOGIN_SERVER/catalog-service:latest"
docker tag order-service:latest "$ACR_LOGIN_SERVER/order-service:latest"
docker tag notification-service:latest "$ACR_LOGIN_SERVER/notification-service:latest"

docker push "$ACR_LOGIN_SERVER/catalog-service:latest"
docker push "$ACR_LOGIN_SERVER/order-service:latest"
docker push "$ACR_LOGIN_SERVER/notification-service:latest"

4. Install the ALB Controller

This Bicep creates a BYO (Bring Your Own) AGC resource along with a User Assigned Managed Identity + Federated Credential + required RBAC for the ALB Controller. On the AKS cluster side, install the ALB Controller with Helm.

Bash:

# Bicep deployment name (default: main-aks)
DEPLOYMENT_NAME="${DEPLOYMENT_NAME:-main-aks}"

# Get the AKS credentials
AKS_NAME=$(az deployment group show -g rg-azure-sre-agent-demo-lab -n "$DEPLOYMENT_NAME" --query properties.outputs.AKS_CLUSTER_NAME.value -o tsv)
ALB_CLIENT_ID=$(az deployment group show -g rg-azure-sre-agent-demo-lab -n "$DEPLOYMENT_NAME" --query properties.outputs.ALB_IDENTITY_CLIENT_ID.value -o tsv)

az aks get-credentials --resource-group rg-azure-sre-agent-demo-lab --name $AKS_NAME

# Install the ALB Controller with Helm
helm upgrade --install alb-controller oci://mcr.microsoft.com/application-lb/charts/alb-controller \
  --namespace azure-alb-system \
  --create-namespace \
  --version 1.9.13 \
  --set albController.namespace=azure-alb-system \
  --set albController.podIdentity.clientID=$ALB_CLIENT_ID

# Verify the installation
kubectl get pods -n azure-alb-system
kubectl get gatewayclass azure-alb-external -o yaml

PowerShell:

# Bicep deployment name (default: main-aks)
$DEPLOYMENT_NAME = if ($DEPLOYMENT_NAME) { $DEPLOYMENT_NAME } else { "main-aks" }

# Get the AKS credentials
$AKS_NAME = (az deployment group show -g rg-azure-sre-agent-demo-lab -n $DEPLOYMENT_NAME --query "properties.outputs.AKS_CLUSTER_NAME.value" -o tsv)
$ALB_CLIENT_ID = (az deployment group show -g rg-azure-sre-agent-demo-lab -n $DEPLOYMENT_NAME --query "properties.outputs.ALB_IDENTITY_CLIENT_ID.value" -o tsv)

az aks get-credentials --resource-group rg-azure-sre-agent-demo-lab --name $AKS_NAME

# Install the ALB Controller with Helm
helm upgrade --install alb-controller oci://mcr.microsoft.com/application-lb/charts/alb-controller `
  --namespace azure-alb-system `
  --create-namespace `
  --version 1.9.13 `
  --set albController.namespace=azure-alb-system `
  --set albController.podIdentity.clientID=$ALB_CLIENT_ID

# Verify the installation
kubectl get pods -n azure-alb-system
kubectl get gatewayclass azure-alb-external -o yaml

See the ALB Controller (Helm) quickstart for details.

5. Kubernetes deployment

Note: Secrets were already registered with the kubectl create secret commands in step 2-5. k8s/secrets.template.yaml is provided as a template, but if you have already registered the secrets, do not apply it (the real values would be overwritten with placeholders). The image fields in k8s/*-service.yaml contain the <ACR_NAME> placeholder, so do not edit the YAML in the repository — substitute into a temporary file and deploy that.

Bash:

./scripts/deploy-workloads-with-acr.sh your-acr-name

PowerShell 7:

./scripts/deploy-workloads-with-acr.ps1 -AcrName your-acr-name

If you prefer not to use the scripts above, you can also deploy with the following steps. However, you must replace <ACR_NAME> with the actual ACR login server name.

Bash:

# Set ACR_NAME
ACR_NAME="your-acr-name"

# Substitute into the manifests and apply
sed "s|<ACR_NAME>|${ACR_NAME}|g" k8s/namespace.yaml | kubectl apply -f -
sed "s|<ACR_NAME>|${ACR_NAME}|g" k8s/catalog-service.yaml | kubectl apply -f -
sed "s|<ACR_NAME>|${ACR_NAME}|g" k8s/order-service.yaml | kubectl apply -f -
sed "s|<ACR_NAME>|${ACR_NAME}|g" k8s/notification-service.yaml | kubectl apply -f -

PowerShell 7:

# Set ACR_NAME
$ACR_NAME = "your-acr-name"

# Substitute into the manifests and apply
(Get-Content k8s/namespace.yaml) -replace '<ACR_NAME>', $ACR_NAME | kubectl apply -f -
(Get-Content k8s/catalog-service.yaml) -replace '<ACR_NAME>', $ACR_NAME | kubectl apply -f -
(Get-Content k8s/order-service.yaml) -replace '<ACR_NAME>', $ACR_NAME | kubectl apply -f -
(Get-Content k8s/notification-service.yaml) -replace '<ACR_NAME>', $ACR_NAME | kubectl apply -f -

6. Deploy the Gateway API resources

Replace <AGC_RESOURCE_ID> and <AGC_FRONTEND_NAME> in k8s/gateway.yaml with the values output by the Bicep deployment, then apply.

Bash:

# Bicep deployment name (default: main-aks)
DEPLOYMENT_NAME="${DEPLOYMENT_NAME:-main-aks}"

# Get the AGC resource ID
AGC_ID=$(az deployment group show -g rg-azure-sre-agent-demo-lab -n "$DEPLOYMENT_NAME" \
  --query properties.outputs.AGC_RESOURCE_ID.value -o tsv)

# Get the AGC frontend name
AGC_FRONTEND_NAME=$(az deployment group show -g rg-azure-sre-agent-demo-lab -n "$DEPLOYMENT_NAME" \
  --query properties.outputs.AGC_FRONTEND_NAME.value -o tsv)

# Substitute the placeholders in gateway.yaml and apply
sed -e "s|<AGC_RESOURCE_ID>|${AGC_ID}|g" \
    -e "s|<AGC_FRONTEND_NAME>|${AGC_FRONTEND_NAME}|g" \
    k8s/gateway.yaml | kubectl apply -f -

# Apply the HTTPRoute
kubectl apply -f k8s/httproutes.yaml

PowerShell:

# Bicep deployment name (default: main-aks)
$DEPLOYMENT_NAME = if ($DEPLOYMENT_NAME) { $DEPLOYMENT_NAME } else { "main-aks" }

# Get the AGC resource ID
$AGC_ID = (az deployment group show -g rg-azure-sre-agent-demo-lab -n $DEPLOYMENT_NAME --query "properties.outputs.AGC_RESOURCE_ID.value" -o tsv)

# Get the AGC frontend name
$AGC_FRONTEND_NAME = (az deployment group show -g rg-azure-sre-agent-demo-lab -n $DEPLOYMENT_NAME --query "properties.outputs.AGC_FRONTEND_NAME.value" -o tsv)

# Substitute the placeholders in gateway.yaml and apply
(Get-Content k8s/gateway.yaml) `
  -replace '<AGC_RESOURCE_ID>', $AGC_ID `
  -replace '<AGC_FRONTEND_NAME>', $AGC_FRONTEND_NAME |
  kubectl apply -f -

# Apply the HTTPRoute
kubectl apply -f k8s/httproutes.yaml

7. Verify

Bash:

# Bicep deployment name (default: main-aks)
DEPLOYMENT_NAME="${DEPLOYMENT_NAME:-main-aks}"

# Check the Gateway status
kubectl get gateway -n azure-sre-agent-demo-lab

# Check the HTTPRoute status
kubectl get httproute -n azure-sre-agent-demo-lab

# Get the AGC frontend FQDN
AGC_FQDN=$(az deployment group show -g rg-azure-sre-agent-demo-lab -n "$DEPLOYMENT_NAME" \
  --query properties.outputs.AGC_FRONTEND_FQDN.value -o tsv)
echo "Endpoint: http://${AGC_FQDN}"

PowerShell:

# Bicep deployment name (default: main-aks)
$DEPLOYMENT_NAME = if ($DEPLOYMENT_NAME) { $DEPLOYMENT_NAME } else { "main-aks" }

# Check the Gateway status
kubectl get gateway -n azure-sre-agent-demo-lab

# Check the HTTPRoute status
kubectl get httproute -n azure-sre-agent-demo-lab

# Get the AGC frontend FQDN
$AGC_FQDN = (az deployment group show -g rg-azure-sre-agent-demo-lab -n $DEPLOYMENT_NAME --query "properties.outputs.AGC_FRONTEND_FQDN.value" -o tsv)
Write-Host "Endpoint: http://$AGC_FQDN"

8. Troubleshooting

Problem: kubectl apply fails with "dial tcp 127.0.0.1:8080: connectex: No connection could be made"

Cause: kubectl is not connected to the AKS cluster.

Resolution:

Bash:

# 1. Check which cluster you are connected to
kubectl config current-context

# 2. If you are not connected to AKS, get the credentials
DEPLOYMENT_NAME="${DEPLOYMENT_NAME:-main-aks}"
AKS_NAME=$(az deployment group show -g rg-azure-sre-agent-demo-lab -n "$DEPLOYMENT_NAME" --query properties.outputs.AKS_CLUSTER_NAME.value -o tsv)
az aks get-credentials --resource-group rg-azure-sre-agent-demo-lab --name $AKS_NAME --overwrite-existing

# 3. Verify the connection
kubectl cluster-info
kubectl get nodes

PowerShell:

# 1. Check which cluster you are connected to
kubectl config current-context

# 2. If you are not connected to AKS, get the credentials
$AKS_NAME = (az deployment group show -g rg-azure-sre-agent-demo-lab -n main-aks --query "properties.outputs.akS_CLUSTER_NAME.value" -o tsv)
az aks get-credentials --resource-group rg-azure-sre-agent-demo-lab --name $AKS_NAME --overwrite-existing

# 3. Verify the connection
kubectl cluster-info
kubectl get nodes

Problem: Cannot retrieve the Cosmos DB connection string

Cause: The resource group name or resource name may be incorrect.

Resolution:

Bash:

# Check the Cosmos DB account name from the Bicep deployment outputs
az deployment group show \
  -g rg-azure-sre-agent-demo-lab \
  -n main-aks \
  --query "properties.outputs.cosmoS_ACCOUNT_NAME.value" -o tsv

# Check the resource groups
az group list --query "[].name" -o table

PowerShell:

# Check the Cosmos DB account name from the Bicep deployment outputs
az deployment group show `
  -g rg-azure-sre-agent-demo-lab `
  -n main-aks `
  --query "properties.outputs.cosmoS_ACCOUNT_NAME.value" -o tsv

# Check the resource groups
az group list --query "[].name" -o table

Problem: Namespace azure-sre-agent-demo-lab does not exist

Resolution:

Bash / PowerShell (same command for both):

# Create the namespace
kubectl create namespace azure-sre-agent-demo-lab

Entra ID authentication flow (AKS environment)

All services deployed to AKS are protected with JWT bearer authentication. Use the steps below to obtain an access token from Entra ID and call the APIs.

Set the prerequisite variables

Bash:

TENANT_ID="<YOUR_TENANT_ID>"   # az account show --query tenantId -o tsv
CLIENT_ID="<YOUR_CLIENT_ID>"   # Application (client) ID of the app registration
CLIENT_SECRET="<YOUR_CLIENT_SECRET>"
API_SCOPE="api://${CLIENT_ID}/access_as_user"
# Get the AGC frontend FQDN
AGC_FQDN=$(kubectl get gateway azure-sre-agent-demo-lab-gateway \
  -n azure-sre-agent-demo-lab \
  -o jsonpath='{.status.addresses[0].value}')
if [ -z "$AGC_FQDN" ]; then
  AGC_FQDN=$(az deployment group show -g rg-azure-sre-agent-demo-lab -n main-aks \
    --query properties.outputs.agC_FRONTEND_FQDN.value -o tsv)
fi

echo "AGC endpoint: ${AGC_FQDN}"
# If empty, check the status with:
# kubectl get gateway azure-sre-agent-demo-lab-gateway -n azure-sre-agent-demo-lab -o wide

PowerShell:

$TENANT_ID = "<YOUR_TENANT_ID>"   # az account show --query tenantId -o tsv
$CLIENT_ID = "<YOUR_CLIENT_ID>"   # Application (client) ID of the app registration
$CLIENT_SECRET = "<YOUR_CLIENT_SECRET>"
$API_SCOPE = "api://$CLIENT_ID/access_as_user"
$DEPLOYMENT_NAME = if ($DEPLOYMENT_NAME) { $DEPLOYMENT_NAME } else { "main-aks" }
# Get the AGC frontend FQDN
$AGC_FQDN = (kubectl get gateway azure-sre-agent-demo-lab-gateway -n azure-sre-agent-demo-lab -o jsonpath='{.status.addresses[0].value}' 2>$null)
if (-not $AGC_FQDN) {
  $AGC_FQDN = (az deployment group show -g rg-azure-sre-agent-demo-lab -n $DEPLOYMENT_NAME --query "properties.outputs.AGC_FRONTEND_FQDN.value" -o tsv)
}

Write-Host "AGC endpoint: $AGC_FQDN"
# If empty, check the status with:
# kubectl get gateway azure-sre-agent-demo-lab-gateway -n azure-sre-agent-demo-lab -o wide

Obtain an access token (client_credentials flow)

Bash:

TOKEN=$(curl -s -X POST \
  "https://login.microsoftonline.com/${TENANT_ID}/oauth2/v2.0/token" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "grant_type=client_credentials" \
  -d "client_id=${CLIENT_ID}" \
  -d "client_secret=${CLIENT_SECRET}" \
  -d "scope=api://${CLIENT_ID}/.default" \
  | jq -r .access_token)

if [ -n "$TOKEN" ]; then
  echo "Token acquired: yes"
else
  echo "Token acquired: no"
fi

If jq is unavailable, replace it with python3 -c "import sys,json; print(json.load(sys.stdin)['access_token'])".

PowerShell:

# Helper function for URL encoding
function ConvertTo-UrlEncoded($str) {
  return [System.Net.WebUtility]::UrlEncode($str)
}

$encodedClientSecret = ConvertTo-UrlEncoded $CLIENT_SECRET
$tokenResponse = Invoke-RestMethod `
  -Method Post `
  -Uri "https://login.microsoftonline.com/$TENANT_ID/oauth2/v2.0/token" `
  -ContentType "application/x-www-form-urlencoded" `
  -Body "grant_type=client_credentials&client_id=$CLIENT_ID&client_secret=$encodedClientSecret&scope=api://$CLIENT_ID/.default"

$TOKEN = $tokenResponse.access_token
if ($TOKEN) { Write-Host "Token acquired: yes" } else { Write-Host "Token acquired: no" }

Obtain an access token (Azure CLI login credentials)

This flow acquires an access token using the user credentials already signed in to the Azure CLI. CLIENT_SECRET is not required.

Prerequisite: the API must expose the delegated scope (access_as_user), and user consent or admin consent must be granted.

Bash:

# Sign in to the Azure CLI (only if not already signed in)
az login --tenant "$TENANT_ID"

# Select the target subscription if needed
az account set --subscription "<YOUR_SUBSCRIPTION_ID>"

TOKEN=$(az account get-access-token \
  --tenant "$TENANT_ID" \
  --scope "$API_SCOPE" \
  --query accessToken -o tsv)

if [ -n "$TOKEN" ]; then
  echo "Token acquired: yes"
else
  echo "Token acquired: no"
fi

PowerShell:

# Sign in to the Azure CLI (only if not already signed in)
az login --tenant $TENANT_ID

# Select the target subscription if needed
az account set --subscription "<YOUR_SUBSCRIPTION_ID>"

$TOKEN = (az account get-access-token `
  --tenant $TENANT_ID `
  --scope $API_SCOPE `
  --query accessToken -o tsv)

if ($TOKEN) { Write-Host "Token acquired: yes" } else { Write-Host "Token acquired: no" }

az account get-access-token returns a user delegation token. For background services or other contexts that run without a user, use the client_credentials flow above.

Example authenticated API calls

Bash:

# List products (CatalogService)
curl -s -H "Authorization: Bearer $TOKEN" \
  http://$AGC_FQDN/catalog/api/products | jq .

# List categories (CatalogService)
curl -s -H "Authorization: Bearer $TOKEN" \
  http://$AGC_FQDN/catalog/api/categories | jq .

# List orders (OrderService)
curl -s -H "Authorization: Bearer $TOKEN" \
  http://$AGC_FQDN/orders/api/orders | jq .

# List notifications (NotificationService)
curl -s -H "Authorization: Bearer $TOKEN" \
  http://$AGC_FQDN/notifications/api/notifications | jq .

Accessing without a token returns HTTP 401 Unauthorized:

curl -o /dev/null -w "%{http_code}" http://$AGC_FQDN/catalog/api/products
# → 401

PowerShell:

$headers = @{ Authorization = "Bearer $TOKEN" }

# List products (CatalogService)
Invoke-RestMethod -Uri "http://$AGC_FQDN/catalog/api/products" -Headers $headers

# List categories (CatalogService)
Invoke-RestMethod -Uri "http://$AGC_FQDN/catalog/api/categories" -Headers $headers

# List orders (OrderService)
Invoke-RestMethod -Uri "http://$AGC_FQDN/orders/api/orders" -Headers $headers

# List notifications (NotificationService)
Invoke-RestMethod -Uri "http://$AGC_FQDN/notifications/api/notifications" -Headers $headers

Accessing without a token returns HTTP 401 Unauthorized:

(Invoke-WebRequest -Uri "http://$AGC_FQDN/catalog/api/products" -SkipHttpErrorCheck).StatusCode
# → 401

Reproducing the bugs in the AKS environment (authenticated)

Premium slow path

Bash:

# Trigger an intentional slowdown with a "premium" query
curl -s -H "Authorization: Bearer $TOKEN" \
  "http://$AGC_FQDN/catalog/api/products/search?q=premium"

PowerShell:

# Trigger an intentional slowdown with a "premium" query
Invoke-RestMethod -Uri "http://$AGC_FQDN/catalog/api/products/search?q=premium" `
  -Headers @{ Authorization = "Bearer $TOKEN" }
  • Expected behavior: the response takes several seconds (a Thread.Sleep loop)
  • Inspecting the trace in OpenTelemetry shows an inflated duration on the SearchAsync span

Long-query 500 error path

Bash:

# Trigger an ArgumentOutOfRangeException with a query longer than 100 characters
LONG_QUERY=$(python3 -c "print('a' * 150)")
curl -s -o /dev/null -w "%{http_code}" \
  -H "Authorization: Bearer $TOKEN" \
  "http://$AGC_FQDN/catalog/api/products/search?q=${LONG_QUERY}"
# → 500

PowerShell:

# Trigger an ArgumentOutOfRangeException with a query longer than 100 characters
$LONG_QUERY = "a" * 150
$response = Invoke-WebRequest `
  -Uri "http://$AGC_FQDN/catalog/api/products/search?q=$LONG_QUERY" `
  -Headers @{ Authorization = "Bearer $TOKEN" } `
  -SkipHttpErrorCheck
$response.StatusCode  # → 500
  • Expected behavior: HTTP 500 is returned
  • An exception log with a stack trace is recorded in Application Insights / OpenTelemetry

Azure SRE Agent Demo Integration

This repository includes a complete, executable 20-minute demo showing how the Azure SRE Agent detects incidents, forwards them through an Azure relay, and uses memory & reasoning to suggest resolutions for the downstream work item or issue.

Quick Start

  1. Setup — Configure SRE Agent resources (5 min)

    Bash:

    az deployment group create \
      --resource-group <resource-group> \
      --template-file infra/sre-overlay.bicep \
      --parameters \
        "environmentName=<environment-name>" \
        "location=<base-resource-location>" \
        "incidentRelayResourceId=<logic-app-resource-id>" \
        "incidentRelayCallbackUrl=<logic-app-callback-url>" \
        "responseTimeThresholdMs=500" \
        "failedRequestCountThreshold=5"

    PowerShell:

    az deployment group create `
      --resource-group <resource-group> `
      --template-file infra/sre-overlay.bicep `
      --parameters `
        "environmentName=<environment-name>" `
        "location=<base-resource-location>" `
        "incidentRelayResourceId=<logic-app-resource-id>" `
        "incidentRelayCallbackUrl=<logic-app-callback-url>" `
        "responseTimeThresholdMs=500" `
        "failedRequestCountThreshold=5"

    incidentRelayCallbackUrl must be the full HTTP trigger callback URL returned by listCallbackUrl, such as https://.../triggers/.../paths/invoke?...&sig=.... Do not use the Logic App overview URL.

    The quotes around each key=value argument are required when the callback URL contains query string parameters such as &sp=, &sv=, and &sig=.

    Note: To update only the SRE resources, use infra/sre-overlay.bicep. Re-running infra/main.bicep after azd deploy may revert the Container Apps to the provisioning sample image (mcr.microsoft.com/dotnet/samples:aspnetapp). If you see the sample image, run azd deploy to re-apply the service images.

  2. Run the Demo — Complete incident cycle (20 min)

Key Features

✅ Azure Monitor Alerts — Auto-detect latency & error rate anomalies ✅ Azure-native Relay Integration — Logic Apps or Azure Functions receive the Action Group payload ✅ Downstream Ticket Routing — Your relay can create Azure DevOps work items, GitHub issues, or both ✅ SRE Agent Memory & Runbooks — Knowledge base for incident response ✅ Agent Reasoning — Suggests root causes with confidence scores ✅ End-to-End Demo — Executable in 20 minutes

Documentation

Architecture

Microservices → OpenTelemetry → Application Insights
    ↓
Log Analytics (30-day retention)
    ↓
Alert Rules (latency, error rate)
    ↓
Action Group → Logic App / Azure Function
    ↓
Configured downstream work item / issue
    ↓
SRE Agent (reads incident + memory + telemetry)
    ↓
Agent Reasoning → Suggests root cause & resolution

Resources Deployed (when enabled)

  • 2 Metric Alert Rules (latency, failed requests)
  • 1 Action Group (routes to an Azure-native relay; the relay decides Azure DevOps, GitHub, or both)
  • 1 Scheduled Query Alert (custom incident detection)
  • Diagnostic Settings on Container Apps Environment

⚠️ Intentional bug (for SRE training)

This application embeds exactly one intentional performance issue for educational purposes.

Location

The SearchAsync method in src/CatalogService/Services/ProductSearchService.cs.

Trigger conditions

  1. Slowdown: Send a request whose query parameter q contains premium.

    curl http://localhost:5001/api/products/search?q=premium
    • Cause: a synchronous loop containing Thread.Sleep(100) processes all products sequentially
    • Result: the response time balloons to several seconds and the thread pool is exhausted
  2. 500 error: Send a string longer than 100 characters in the query parameter q.

    curl "http://localhost:5001/api/products/search?q=$(python3 -c 'print("a"*150)')"
    • Cause: a missing boundary check in String.Substring
    • Result: ArgumentOutOfRangeException → HTTP 500

Sample data

Automatically seeded into Cosmos DB when each service starts:

Data Count Description
Products 20 Electronics, clothing, food, books, premium
Categories 5 Electronics, Clothing, Food, Books, Premium
Inventory 20 One per product
Orders 10 Various statuses
Notifications 15 Order confirmations, shipping notices, promotions, etc.

Note: The Cosmos DB database name GlobalAzureDemo is a legacy runtime/data-plane value and is intentionally retained even after the project was renamed to AzureSREAgentDemoLab.

License

This project is licensed under the MIT License.

About

for Azure SRE Agent and Azure Copilot observerbility Agent demo repo

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages