[BRE-1413] Investigation for supporting Canary via header based routing

CANARY-FIXES.md

@@ -0,0 +1,44 @@
+# Canary Header Propagation Changes
+
+## What this does
+
+Three things are needed for coordinated canary deployments where all canary pods route to other canary pods:
+
+1. **Header propagation** — When a request arrives with a configured header (e.g. `X-Canary: true`), that header must be forwarded on all outbound HTTP calls so downstream services are also routed to their canary pods via AGC. Uses .NET 8's built-in `HeaderPropagation` middleware + `ConfigureHttpClientDefaults` to cover all `HttpClient` instances globally.
+
+2. **OIDC backchannel fix** — The Identity service's OpenID Connect handler creates its own internal `Backchannel` HttpClient that bypasses `IHttpClientFactory`. This is fixed by explicitly wiring the OIDC handler to use a factory-created client, ensuring the propagation handler applies to Identity → SSO backchannel calls.
+
+3. **Response middleware** — Echoes back a routed confirmation header (e.g. `X-Canary-Routed: true`) and tags the active trace span with pod name, build hash, and canary status for DataDog observability.
+
+All are **configuration-driven** — the header name is set via ConfigMap (`HeaderPropagation__Headers__0=X-Canary`), never hardcoded in server code. When unconfigured (e.g. self-hosted), all features are no-ops.
+
+## Service coverage
+
+| Service | Outbound propagation | Inbound middleware | Notes |
+|---|---|---|---|
+| **API** | `AddDefaultServices` | `UseDefaultMiddleware` | Makes calls to Identity, Notifications |
+| **Identity** | `AddDefaultServices` | `UseDefaultMiddleware` | Makes calls to SSO (OIDC backchannel also fixed) |
+| **Billing** | `AddDefaultServices` | `UseDefaultMiddleware` (added) | Receives calls from Admin |
+| **Admin** | `AddDefaultServices` | `UseDefaultMiddleware` (added) | Makes calls to Billing, Identity |
+| **SSO** | `AddDefaultServices` | `UseDefaultMiddleware` (added) | Receives calls from Identity |
+| **SCIM** | `AddDefaultServices` | `UseDefaultMiddleware` (added) | Makes calls to Identity |
+| **Events** | N/A | `UseDefaultMiddleware` | No outbound inter-service calls |
+| **Notifications** | N/A | `UseDefaultMiddleware` (added) | No outbound inter-service calls |
+| **Icons** | N/A | `UseDefaultMiddleware` (added) | No outbound inter-service calls |
+| **EventsProcessor** | N/A | N/A | Background worker, no HTTP pipeline |
+
+Services marked "N/A" for outbound propagation don't make inter-service HTTP calls, so they only need the inbound middleware for the response header and DataDog tagging. Services marked "(added)" did not previously call that method and were updated in this change.
+
+`UseDefaultMiddleware` conditionally calls `UseHeaderPropagation()` only if `AddHeaderPropagation()` was registered (via `AddDefaultServices`), preventing runtime errors on services that don't make outbound HTTP calls.
+
+## Shared resources (not isolated by canary)
+
+Some communication channels between services are not HTTP-based. Header propagation does not and cannot isolate these. This is by design — canary routing operates at the HTTP layer only.
+
+**Database** — Canary and stable pods read from and write to the same database. A prerequisite for canary deployments is that database migrations are decoupled from application deployments: both the current and next app versions must handle the current and next database schema. This is being validated independently.
+
+**Azure Service Bus** — Events and application cache messages are published to shared topics/subscriptions. A canary pod may publish a message consumed by a stable pod and vice versa. Both app versions must handle messages from either version.
+
+**Redis** — The SignalR backplane uses shared Redis. Notifications published by canary pods may be relayed through stable pod connections. Both app versions must handle notifications from either version.
+
+These constraints apply equally to today's rolling deployments where old and new pods coexist during rollout. Canary does not introduce new data-plane risks — it extends the coexistence window.

CANARY-ISSUES.md

@@ -0,0 +1,54 @@
+# Canary Header Propagation Issues
+
+Without header propagation, inter-service calls from canary pods lose the `X-Canary` header and fall through to stable pods via AGC. This breaks canary isolation in a coordinated deployment where all services should stay within the canary call chain.
+
+## Issue 1: Inter-service HTTP calls lose the canary header
+
+All cloud inter-service calls use external HTTPS URLs (e.g. `https://identity.usdev.bitwarden.pw`) which route through AGC. Without the `X-Canary` header on these outbound calls, AGC routes them to stable pods.
+
+**Affected calls:**
+
+**API → Identity** (token acquisition)
+- `BaseIdentityClientService` calls `POST /connect/token` on Identity to get bearer tokens
+- Every service that makes authenticated inter-service calls hits this path first
+
+**API → Notifications** (push notifications)
+- `NotificationsApiPushEngine` calls `POST /send` on the Notifications service
+- Triggered on cipher operations, folder changes, org updates — high frequency
+
+**Admin → Billing** (Stripe recovery)
+- `ProcessStripeEventsController` calls `POST /stripe/recovery/events/*` on Billing
+- Lower frequency, but would still route to stable Billing during canary
+
+**Admin → Identity** (token acquisition)
+- Same `BaseIdentityClientService` pattern as API
+
+**Identity → SSO** (OIDC)
+- Named HttpClient `"InternalSso"` in Identity's Startup
+- Any SSO login flow would route to stable SSO
+
+**Notifications → Identity** (token acquisition)
+- Same `BaseIdentityClientService` token pattern
+
+**Billing → Identity** (token acquisition)
+- Same `BaseIdentityClientService` token pattern
+
+**Highest risk:** API → Identity and API → Notifications are the most frequent calls. A canary API pod would get its token from stable Identity and send push notifications to stable Notifications — breaking the canary isolation chain.
+
+## Issue 2: OIDC backchannel bypasses HttpClientFactory
+
+The Identity service's OpenID Connect middleware (for SSO login flows) creates its own internal `Backchannel` HttpClient. This client is not created through `IHttpClientFactory`, so `ConfigureHttpClientDefaults` does not apply. The `X-Canary` header would not propagate on Identity → SSO backchannel calls (discovery, token exchange, userinfo).
+
+## Issue 3: Shared resources are not isolated by canary routing
+
+Some communication channels between services are not HTTP and cannot be isolated via header propagation:
+
+- **Database** — Canary and stable pods share the same database. Writes from canary pods are visible to stable pods and vice versa.
+- **Azure Service Bus** — Events and application cache messages are published to shared topics. A canary pod may publish a message consumed by a stable pod.
+- **Redis** — SignalR backplane uses shared Redis. Notifications published by canary pods may be relayed by stable pods.
+
+These are data-plane concerns, not routing concerns. Header propagation cannot address them.
+
+## Resolution
+
+See [CANARY-FIXES.md](CANARY-FIXES.md) for how Issues 1 and 2 are resolved. Issue 3 is addressed by design constraints documented there.

bitwarden_license/src/Scim/Startup.cs

@@ -69,7 +69,7 @@ public void ConfigureServices(IServiceCollection services)
 
         // Services
         services.AddBaseServices(globalSettings);
-        services.AddDefaultServices(globalSettings);
+        services.AddDefaultServices(globalSettings, Configuration);
         services.AddDistributedCache(globalSettings);
         services.AddBillingOperations();
 
@@ -112,7 +112,7 @@ public void Configure(
         }
 
         // Default Middleware
-        app.UseDefaultMiddleware(env, globalSettings);
+        app.UseDefaultMiddleware(env, globalSettings, Configuration);
 
         // Add routing
         app.UseRouting();

bitwarden_license/src/Sso/Startup.cs

@@ -79,7 +79,7 @@ public void ConfigureServices(IServiceCollection services)
 
         // Services
         services.AddBaseServices(globalSettings);
-        services.AddDefaultServices(globalSettings);
+        services.AddDefaultServices(globalSettings, Configuration);
         services.AddCoreLocalizationServices();
         services.AddBillingOperations();
 
@@ -124,6 +124,8 @@ public void Configure(
             app.UseExceptionHandler("/Error");
         }
 
+        app.UseDefaultMiddleware(environment, globalSettings, Configuration);
+
         app.UseCoreLocalization();
 
         // Add static files to the request pipeline.

src/Admin/Startup.cs

@@ -87,7 +87,7 @@ public void ConfigureServices(IServiceCollection services)
 
         // Services
         services.AddBaseServices(globalSettings);
-        services.AddDefaultServices(globalSettings);
+        services.AddDefaultServices(globalSettings, Configuration);
         services.AddScoped<IAccessControlService, AccessControlService>();
         services.AddDistributedCache(globalSettings);
         services.AddBillingOperations();
@@ -144,6 +144,9 @@ public void Configure(
             app.UseForwardedHeaders(globalSettings);
         }
 
+        // Default Middleware
+        app.UseDefaultMiddleware(env, globalSettings, Configuration);
+
         if (env.IsDevelopment())
         {
             app.UseDeveloperExceptionPage();

src/Api/Startup.cs

@@ -181,7 +181,7 @@ public void ConfigureServices(IServiceCollection services)
 
         // Services
         services.AddBaseServices(globalSettings);
-        services.AddDefaultServices(globalSettings);
+        services.AddDefaultServices(globalSettings, Configuration);
         services.AddOrganizationSubscriptionServices();
         services.AddCoreLocalizationServices();
         services.AddBillingOperations();
@@ -241,7 +241,7 @@ public void Configure(
         app.UseMiddleware<SecurityHeadersMiddleware>();
 
         // Default Middleware
-        app.UseDefaultMiddleware(env, globalSettings);
+        app.UseDefaultMiddleware(env, globalSettings, Configuration);
 
         if (!globalSettings.SelfHosted)
         {

src/Billing/Startup.cs

@@ -9,6 +9,7 @@
 using Bit.Core.Context;
 using Bit.Core.SecretsManager.Repositories;
 using Bit.Core.SecretsManager.Repositories.Noop;
+using Bit.Core.Settings;
 using Bit.Core.Utilities;
 using Bit.SharedWeb.Utilities;
 using Microsoft.Extensions.DependencyInjection.Extensions;
@@ -80,7 +81,7 @@ public void ConfigureServices(IServiceCollection services)
 
         // Services
         services.AddBaseServices(globalSettings);
-        services.AddDefaultServices(globalSettings);
+        services.AddDefaultServices(globalSettings, Configuration);
         services.AddDistributedCache(globalSettings);
         services.AddBillingOperations();
         services.AddCommercialCoreServices();
@@ -123,11 +124,15 @@ public void ConfigureServices(IServiceCollection services)
 
     public void Configure(
         IApplicationBuilder app,
-        IWebHostEnvironment env)
+        IWebHostEnvironment env,
+        GlobalSettings globalSettings)
     {
         // Add general security headers
         app.UseMiddleware<SecurityHeadersMiddleware>();
 
+        // Default Middleware
+        app.UseDefaultMiddleware(env, globalSettings, Configuration);
+
         if (env.IsDevelopment())
         {
             app.UseDeveloperExceptionPage();

src/Events/Startup.cs

@@ -107,7 +107,7 @@ public void Configure(
         }
 
         // Default Middleware
-        app.UseDefaultMiddleware(env, globalSettings);
+        app.UseDefaultMiddleware(env, globalSettings, Configuration);
 
         // Add routing
         app.UseRouting();

src/Icons/Startup.cs

@@ -71,6 +71,9 @@ public void Configure(
             app.UseForwardedHeaders(globalSettings);
         }
 
+        // Default Middleware
+        app.UseDefaultMiddleware(env, globalSettings, Configuration);
+
         if (env.IsDevelopment())
         {
             app.UseDeveloperExceptionPage();

src/Identity/Startup.cs

@@ -145,7 +145,7 @@ public void ConfigureServices(IServiceCollection services)
 
         // Services
         services.AddBaseServices(globalSettings);
-        services.AddDefaultServices(globalSettings);
+        services.AddDefaultServices(globalSettings, Configuration);
         services.AddOptionality();
         services.AddCoreLocalizationServices();
         services.AddBillingOperations();
@@ -165,6 +165,14 @@ public void ConfigureServices(IServiceCollection services)
         {
             client.BaseAddress = new Uri(globalSettings.BaseServiceUri.InternalSso);
         });
+
+        // Wire the OIDC "sso" handler to use a factory-created HttpClient so that
+        // ConfigureHttpClientDefaults (header propagation) applies to backchannel calls.
+        services.AddOptions<Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectOptions>("sso")
+            .Configure<IHttpClientFactory>((options, factory) =>
+            {
+                options.Backchannel = factory.CreateClient("InternalSso");
+            });
     }
 
     public void Configure(
@@ -193,7 +201,7 @@ public void Configure(
         }
 
         // Default Middleware
-        app.UseDefaultMiddleware(environment, globalSettings);
+        app.UseDefaultMiddleware(environment, globalSettings, Configuration);
 
         if (!globalSettings.SelfHosted)
         {

src/Notifications/Startup.cs

@@ -92,6 +92,9 @@ public void Configure(
             app.UseForwardedHeaders(globalSettings);
         }
 
+        // Default Middleware
+        app.UseDefaultMiddleware(env, globalSettings, Configuration);
+
         if (env.IsDevelopment())
         {
             app.UseDeveloperExceptionPage();

src/SharedWeb/SharedWeb.csproj

@@ -5,13 +5,18 @@
     <WarningsNotAsErrors>$(WarningsNotAsErrors);CA1304</WarningsNotAsErrors>
   </PropertyGroup>
 
+  <ItemGroup>
+    <FrameworkReference Include="Microsoft.AspNetCore.App" />
+  </ItemGroup>
+
   <ItemGroup>
     <ProjectReference Include="..\Infrastructure.Dapper\Infrastructure.Dapper.csproj" />
     <ProjectReference Include="..\Core\Core.csproj" />
     <ProjectReference Include="..\Infrastructure.EntityFramework\Infrastructure.EntityFramework.csproj" />
   </ItemGroup>
 
   <ItemGroup>
+    <PackageReference Include="Microsoft.AspNetCore.HeaderPropagation" Version="8.0.11" />
     <PackageReference Include="Microsoft.Bot.Builder.Integration.AspNet.Core" Version="4.23.0" />
     <PackageReference Include="Swashbuckle.AspNetCore.SwaggerGen" Version="10.1.7" />
   </ItemGroup>

src/SharedWeb/Utilities/HeaderPropagationResponseMiddleware.cs

@@ -0,0 +1,44 @@
+using System.Diagnostics;
+using Bit.Core.Utilities;
+using Microsoft.AspNetCore.Http;
+
+namespace Bit.SharedWeb.Utilities;
+
+/// <summary>
+/// Middleware that adds response headers and telemetry tags for propagated headers.
+/// When a configured propagation header is present on the inbound request,
+/// this middleware sets a corresponding response header (e.g. X-Canary → X-Canary-Routed: true)
+/// and tags the active trace span for observability (DataDog/OpenTelemetry).
+/// </summary>
+public class HeaderPropagationResponseMiddleware
+{
+    private static readonly string _podName = Environment.GetEnvironmentVariable("HOSTNAME") ?? "unknown";
+    private static readonly string _gitHash = AssemblyHelpers.GetGitHash() ?? "unknown";
+    private readonly RequestDelegate _next;
+    private readonly string[] _headers;
+
+    public HeaderPropagationResponseMiddleware(RequestDelegate next, string[] headers)
+    {
+        _next = next;
+        _headers = headers;
+    }
+
+    public async Task InvokeAsync(HttpContext context)
+    {
+        var activity = Activity.Current;
+        activity?.SetTag("pod.name", _podName);
+        activity?.SetTag("build.hash", _gitHash);
+
+        foreach (var header in _headers)
+        {
+            if (context.Request.Headers.TryGetValue(header, out var value)
+                && string.Equals(value, "true", StringComparison.OrdinalIgnoreCase))
+            {
+                activity?.SetTag(header.TrimStart('X', '-').ToLowerInvariant(), "true");
+                context.Response.Headers[$"{header}-Routed"] = "true";
+            }
+        }
+
+        await _next(context);
+    }
+}