Skip to content
Draft
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
177 changes: 177 additions & 0 deletions decisions/012-networking-http2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
# ADR: Unified Networking and HTTP/2 Support

## Context

**Problem**: RHDH frontend startup time grows significantly with larger plugin counts (40+ plugins). The number of frontend assets (JavaScript bundles, CSS) increases with each plugin, and the Backstage New Frontend System (NFS) further increases asset count. With HTTP/1.1, browsers are limited to ~6 concurrent connections per domain, causing sequential loading and slower page render times.

@rm3l rm3l Jul 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With HTTP/1.1, browsers are limited to ~6 concurrent connections per domain, causing sequential loading and slower page render times.

I feel like this could be misleading as it reads like the limit is due to HTTP/1.1. The ~6 concurrent connection limit is a browser-imposed limit, not an HTTP/1.1 protocol constraint. Browsers apply this limit regardless of HTTP version, and the limit is set per domain. With HTTP/2, the connection limit is still there, except that browsers can typically send multiple requests to a same domain over one connection.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is misleading?
In a context of proposing HTTP/2 as a workaround we want to bypass the "limitation" of HTTP/1.1 and browser.
Explaining how exactly browser works is out of the scope of this ADR and it is clear IMO that we do not try to bypass browser limitation, we try to use potential of HTTP/2 protocol, no?


HTTP/2 provides significant performance benefits through:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also clarify that we are talking about HTTP/2 on the public-facing connection segment (i.e, from client to proxy, as highlighted in the POC), not full HTTP/2 end-to-end requiring HTTP/2 also between the proxy/router and the application container (separate concern).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it is clear from the problem statement, the problem we are solving is "frontend startup time". Propose your addition if needed.

- **Multiplexing**: Multiple requests over a single TCP connection
- **Header compression**: Reduced overhead with HPACK
- **No head-of-line blocking**: Independent streams at HTTP level

**Platform differences:**

**OpenShift** has significant constraints:
1. **Cluster-admin requirement**: HTTP/2 must be enabled at the IngressController level:
Comment thread
gazarenkov marked this conversation as resolved.
Outdated
```sh
oc annotate ingresses.config/cluster ingress.operator.openshift.io/default-enable-http2=true
```
2. **Custom certificate requirement**: OpenShift blocks HTTP/2 for routes using the default wildcard certificate (`*.apps.cluster.com`) to prevent connection coalescing issues.
3. **No per-route control**: There is no route-level annotation to enable HTTP/2; it's a cluster-wide decision.

This means RHDH users on shared OpenShift clusters cannot enable HTTP/2 without cluster-admin cooperation.

**Vanilla Kubernetes** is simpler — most Ingress controllers (NGINX, Traefik, HAProxy) support HTTP/2 by default when TLS is enabled. No special configuration required.
Comment thread
gazarenkov marked this conversation as resolved.
Outdated

## Decision

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gazarenkov FYI (also shared during the standup call today), I raised the concerns here in the SOS call yesterday, and we agreed on the following next steps:

  • explicitly document the requirements for users to enable HTTP/2; so this won't be enabled by default, but users will be aware of what to do to improve performance if needed.
  • explicitly claim support for this setup, by essentially testing it on our supported platforms, as this was a concern raised by the customer in the support case

So I think we may no longer need this ADR for now, as we can "just" test and document the setup that was shared in the POC, so we can claim support for it, without having to update the install methods for now. I'll update the Epics under that Feature to reflect that.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ADR did its work - making it clear that initial requirement about HTTP/2 by default does not seem to be realistic and so we can fallback to the simplest possible HTTP/2 solution to try to workaround the frontend performance problem.
This is kinda cool and new option of ADR cycle.
IMO what we need to do is document here that requirements changed and => new decision made (does not matter if it is related to new functionality or "just" testing and documentation, it is still a decision). Current decision should be moved to another Alternative with corresponding rejection reason.


Introduce unified networking configuration for both OpenShift (Route) and vanilla Kubernetes (Ingress), with HTTP/2 proxy support:

- **OpenShift**: Provide optional NGINX sidecar proxy that handles TLS termination and HTTP/2, allowing users to enable HTTP/2 without cluster-admin privileges
- **Vanilla Kubernetes**: Add Ingress configuration; most controllers support HTTP/2 natively with TLS, sidecar optional

**Implementation approach**:

1. **Add optional NGINX sidecar container** to Backstage deployment:
Comment thread
gazarenkov marked this conversation as resolved.
Outdated
```yaml
containers:
- name: backstage
image: backstage:latest
ports:
- containerPort: 7007

- name: http2-proxy
image: nginx:alpine
ports:
- containerPort: 8443
volumeMounts:
- name: tls
mountPath: /etc/nginx/tls
- name: nginx-config
mountPath: /etc/nginx/conf.d
```

2. **NGINX configuration with HTTP/2**:
```nginx
server {
listen 8443 ssl http2;
ssl_certificate /etc/nginx/tls/tls.crt;
ssl_certificate_key /etc/nginx/tls/tls.key;

location / {
proxy_pass http://localhost:7007;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
```

3. **Use passthrough termination** (when http2Proxy enabled):

**OpenShift Route:**
```yaml
apiVersion: route.openshift.io/v1
kind: Route
spec:
tls:
termination: passthrough

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There might be understated consequences to switching from edge to passthrough termination. Could be stated under the consequences section?
From Claude, this is a significant operational change:

  • OpenShift no longer manages certificates for the route. With edge termination, the default wildcard cert or the route's configured certificate is managed by the router. With passthrough, the sidecar owns TLS entirely.
  • HAProxy-level HTTP headers are lost. The router no longer injects X-Forwarded-For, X-Forwarded-Proto, X-Forwarded-Port, etc. The sidecar must be configured to add these, or the Backstage application won't see the original client IP and protocol.
  • Monitoring and logging changes. HAProxy access logs and metrics no longer reflect individual HTTP requests for passthrough routes; they only see TCP connections.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's looks like truth :) so what must we do with this information?

port:
targetPort: https
```

**Kubernetes Ingress** (if http2Proxy used): Configure Ingress controller for SSL passthrough, or use standard TLS termination since most controllers support HTTP/2 natively.

4. **Certificate provisioning options**:
- OpenShift service serving certificates (annotation-based, auto-rotated, but causes browser warnings — internal CA)
- cert-manager with Let's Encrypt (public CA, no browser warnings)
- User-provided certificates via Secret

5. **Expose via `spec.network`**:

**OpenShift (Route):**
```yaml
spec:
network:
route:
enabled: true
host: my-backstage.example.com
tls:
externalCertificateSecretName: my-tls
http2Proxy:
enabled: true # required for HTTP/2 on OpenShift without cluster-admin
```

**Vanilla Kubernetes (Ingress):**
```yaml
spec:
network:
ingress:
enabled: true
host: my-backstage.example.com
className: nginx # optional
tls:
secretName: my-tls
http2Proxy:
enabled: false # usually not needed — most Ingress controllers support HTTP/2 natively
```

- `spec.network.route` — moved from `spec.application.route` (deprecated, supported with warning)
- `spec.network.ingress` — new, for vanilla Kubernetes deployments
- `spec.network.http2Proxy.enabled` — adds NGINX sidecar; switches Route to passthrough / configures Ingress for SSL passthrough
- Certificate reuse: when `http2Proxy.enabled`, proxy uses `route.tls` or `ingress.tls` certificate; defaults to service serving certificates if not specified

**Note:** On vanilla Kubernetes, most Ingress controllers (NGINX, Traefik, HAProxy) support HTTP/2 by default when TLS is enabled. The `http2Proxy` sidecar is primarily needed for OpenShift where cluster-admin controls HTTP/2 at the IngressController level.
Comment thread
gazarenkov marked this conversation as resolved.
Outdated

6. **Default behavior considerations**:

Enabling `http2Proxy` by default is possible but has UX trade-offs:
- **Without user-provided certificate**: Uses OpenShift service serving certificates (internal CA) — HTTP/2 works but browsers show certificate warning

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HTTP/2 works but browsers show certificate warning

Well, for clarity, HTTP/2 won't work until the user explicitly bypasses the certificate error in their browser, as ALPN negotiation happens during the TLS handshake itself.
As currently stated here, it sounds like a minor inconvenience, but in reality, without a user-provided certificate that matches the external RHDH hostname and is signed by a trusted CA (either internal or external to the company), the deployment is broken for browsers in any practical sense. I doubt any enterprise deployment would accept this IMO.

- **With user-provided certificate**: No warnings, full HTTP/2 benefits

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should also clarify that the certificate should be trusted by browsers and match the RHDH hostname. Otherwise, they could provide a self-signed cert or one for the wrong hostname and would still get the same previous hard error in the browser, I guess.


**Options:**
- `http2Proxy.enabled: false` by default — users opt-in, no surprises
- `http2Proxy.enabled: true` by default — HTTP/2 out of the box, but certificate warning unless user provides `route.tls.externalCertificateSecretName`

If decided to default to `enabled: true`: provide clear documentation that users should supply their own certificate to avoid browser warnings. The performance benefit may justify the default, and the warning serves as a signal to configure properly.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If decided to default to enabled: true: provide clear documentation that users should supply their own certificate to avoid browser warnings. The performance benefit may justify the default, and the warning serves as a signal to configure properly.

The JIRA requirement is clear and this was my understanding as well: Enable HTTP/2 by default and Existing deployments gain HTTP/2 on upgrade without manual configuration changes. But I am concerned here about the TLS requirement. Defaulting to enabled would require users to provide certs and proper TLS configuration; this implies that the "upgrade without manual configuration" isn't really do-able here. We need to raise this.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, that's the point.
If we are not able to avoid manual configuration it seems to be nice but less important feature


## Alternatives Considered

@rm3l rm3l Jul 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks there is another possible alternative: Gateway API (GA in OCP and available in vanilla K8s). It seems to rely on Envoy on OCP and h2 works automatically via ALPN without any cluster-admin annotation. However, I've quickly checked on a ROSA 4.21 cluster that while the CRDs are installed, no GatewayClass is deployed by default; so a cluster-admin would still need to set up the Gateway infrastructure first. On vanilla K8s, Gateway API is a standard with multiple implementations but also requires installation. Worth mentioning as a forward-looking alternative with these caveats..

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought there was an earlier revision of the ADR mentioning "Node.js native HTTP/2", or did I hallucinate? :)
Might be worth keeping it as alternative with clear reasoning why it is rejected, no?

@gazarenkov gazarenkov Jul 24, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can mention but I do not think it is about proxy so I did not yhink about it


### Alternative 1: Document cluster-admin HTTP/2 enablement
- **Approach**: Document how cluster-admins can enable HTTP/2 and require custom certificates per-route
- **Rejected because**: Users on shared clusters have no control; requires coordination with cluster-admin for each deployment; doesn't solve the core user autonomy problem

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is true, but might also be the cleanest solution when cluster-admin cooperation is available: no sidecar, no TLS shift, no extra container..

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might but does not fit initial requirements (no admin privileges :) )


### Alternative 2: Standalone HTTP/2 proxy Deployment
- **Approach**: Deploy NGINX/Envoy as separate Deployment + Service; Route/Ingress points to proxy, proxy routes to Backstage Service
- **Rejected because**: Single point of failure; extra network hop adds latency; doesn't scale automatically with Backstage replicas; more complex Service topology to manage

### Alternative 3: Modify Backstage/Node.js to serve HTTP/2 directly
- **Approach**: Configure Backstage's Node.js server to handle HTTP/2 and TLS
- **Rejected because**: Requires upstream Backstage changes; Node.js HTTP/2 is more complex than NGINX; certificate handling in Node.js is less mature

## Consequences

### Positive
✅ Users can enable HTTP/2 without cluster-admin privileges
✅ Faster frontend page loads (multiplexing, reduced connections)
✅ Works on any OpenShift/Kubernetes cluster
✅ NGINX is battle-tested, lightweight (~10MB RAM), and well-understood
Comment thread
gazarenkov marked this conversation as resolved.
Outdated
✅ Optional feature - users who don't need HTTP/2 are unaffected
✅ Works correctly with multi-replica deployments (sidecar per pod)

### Negative
❌ Additional container per pod (slight resource overhead)
❌ Users must manage TLS certificates (unless using service serving certs)
❌ More complex deployment architecture to understand/debug
❌ NGINX configuration must be maintained by operator

### Neutral
⚖️ Changes Route from `edge` to `passthrough` termination when enabled (Ingress: depends on controller)
⚖️ HTTP/2 benefit depends on network conditions (may not help on lossy networks)
⚖️ Browser DevTools needed to verify HTTP/2 is active (Protocol column)

## References

- [Red Hat Blog: gRPC or HTTP/2 Ingress Connectivity in OpenShift](https://www.redhat.com/en/blog/grpc-or-http/2-ingress-connectivity-in-openshift)
Comment thread
gazarenkov marked this conversation as resolved.
- [OpenShift Docs: Configuring Routes](https://docs.openshift.com/container-platform/4.14/networking/routes/route-configuration.html)
Comment thread
gazarenkov marked this conversation as resolved.
Outdated
- [HTTP/2 connection coalescing explanation](https://daniel.haxx.se/blog/2016/08/18/http2-connection-coalescing/)
Comment thread
gazarenkov marked this conversation as resolved.
Loading