Skip to content

ADR-012: Unified Networking and HTTP/2 Support #20

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
gazarenkov wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

ADR-012: Unified Networking and HTTP/2 Support #20

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
83f7ca5
ADR-012 initial
gazarenkov Jul 10, 2026
73179e1
Update decisions/012-networking-http2.md
gazarenkov Jul 13, 2026
e4759d8
fixes and POC
gazarenkov Jul 14, 2026
a4e1bf0
Apply suggestion from @rm3l
gazarenkov Jul 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
148 changes: 148 additions & 0 deletions decisions/012-networking-http2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
# ADR: Unified Networking and HTTP/2 Support

## Context

**Problem**: RHDH frontend startup time grows significantly with larger plugin counts (40+ plugins). The number of frontend assets (JavaScript bundles, CSS) increases with each plugin, and the Backstage New Frontend System (NFS) further increases asset count. With HTTP/1.1, browsers are limited to ~6 concurrent connections per domain, causing sequential loading and slower page render times.

@rm3l rm3l Jul 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With HTTP/1.1, browsers are limited to ~6 concurrent connections per domain, causing sequential loading and slower page render times.

I feel like this could be misleading as it reads like the limit is due to HTTP/1.1. The ~6 concurrent connection limit is a browser-imposed limit, not an HTTP/1.1 protocol constraint. Browsers apply this limit regardless of HTTP version, and the limit is set per domain. With HTTP/2, the connection limit is still there, except that browsers can typically send multiple requests to a same domain over one connection.

@gazarenkov gazarenkov Jul 24, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is misleading?
In a context of proposing HTTP/2 as a workaround we want to bypass the "limitation" of HTTP/1.1 and browser.
Explaining how exactly browser works is out of the scope of this ADR and it is clear IMO that we do not try to bypass browser limitation, we try to use potential of HTTP/2 protocol, no?


HTTP/2 provides significant performance benefits through:

@rm3l rm3l Jul 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also clarify that we are talking about HTTP/2 on the public-facing connection segment (i.e, from client to proxy, as highlighted in the POC), not full HTTP/2 end-to-end requiring HTTP/2 also between the proxy/router and the application container (separate concern).

@gazarenkov gazarenkov Jul 24, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it is clear from the problem statement, the problem we are solving is "frontend startup time". Propose your addition if needed.

- **Multiplexing**: Multiple requests over a single TCP connection
- **Header compression**: Reduced overhead with HPACK
- **No head-of-line blocking**: Independent streams at HTTP level

**Platform constraints:**

**OpenShift:**
- HTTP/2 requires cluster-admin to enable at IngressController level (cluster-wide)
- Custom certificate required (default wildcard certificate blocks HTTP/2 due to connection coalescing)
- No per-route control

**Vanilla Kubernetes:**
- **Traefik**: HTTP/2 enabled by default with TLS
- **Caddy**: HTTP/2 enabled by default with TLS
- **NGINX Ingress Controller**: HTTP/2 disabled by default, requires cluster-admin to enable via ConfigMap (cluster-wide)
- **HAProxy**: Depends on configuration

On both platforms, RHDH users on shared clusters cannot enable HTTP/2 without cluster-admin cooperation.

@rm3l rm3l Jul 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I think this requires further clarification. Users can force-use HTTP/2 even without cluster-admin cooperation. With curl, you can skip the protocol negotiation (ALPN) and talk directly via HTTP/2, and this works, even on OCP without the cluster-admin annotation:

❯ curl -k --http2-prior-knowledge -v -sS -o /dev/null https://console-openshift-console.apps.bcca4ad0ec252f076ae6.hypershift.aws-2.ci.openshift.org/
* Host console-openshift-console.apps.bcca4ad0ec252f076ae6.hypershift.aws-2.ci.openshift.org:443 was resolved.
* IPv6: (none)
* IPv4: 3.209.242.225, 52.200.71.79
*   Trying 3.209.242.225:443...
* ALPN: curl offers h2
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [1635 bytes data]
* SSL Trust: peer verification disabled
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [1210 bytes data]
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [10 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [1038 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519MLKEM768 / RSASSA-PSS
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*   subject: O=openshift; CN=openshift-ingress
*   start date: Jul 15 07:42:27 2026 GMT
*   expire date: Jul 15 07:42:27 2027 GMT
*   issuer: OU=openshift; CN=root-ca
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*  SSL certificate verification failed, continuing anyway!
* Established connection to console-openshift-console.apps.bcca4ad0ec252f076ae6.hypershift.aws-2.ci.openshift.org (3.209.242.225 port 443) from 192.168.1.27 port 38278
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://console-openshift-console.apps.bcca4ad0ec252f076ae6.hypershift.aws-2.ci.openshift.org/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: console-openshift-console.apps.bcca4ad0ec252f076ae6.hypershift.aws-2.ci.openshift.org]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.18.0]
* [HTTP/2] [1] [accept: */*]
} [5 bytes data]
> GET / HTTP/2
> Host: console-openshift-console.apps.bcca4ad0ec252f076ae6.hypershift.aws-2.ci.openshift.org
> User-Agent: curl/8.18.0
> Accept: */*
>
* Request completely sent off
} [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [313 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [313 bytes data]
< HTTP/2 200
< content-security-policy-report-only: base-uri 'self'; default-src 'self' console.redhat.com; img-src 'self' data:; font-src 'self' data:; script-src 'self' console.redhat.com 'unsafe-eval' 'nonce-3qZBYQoZ9PCRu1xyhfPKqbGkYIZZAJQL'; style-src 'self' 'unsafe-inline'; connect-src 'self' console.redhat.com; object-src 'self'; frame-src 'none'; frame-ancestors 'none'
< referrer-policy: strict-origin-when-cross-origin
< set-cookie: csrf-token=sV9OZyeKdRSNZ5jRhkqjiTLM2NuM6Nou3NwxQ0ygH2PNoaPDDaZsPglZP4Akg7HO; Path=/; Secure; SameSite=Strict
< x-content-type-options: nosniff
< x-dns-prefetch-control: off
< x-frame-options: DENY
< date: Wed, 15 Jul 2026 10:28:41 GMT
< content-type: text/html; charset=utf-8
< set-cookie: 1e2670d92730b515ce3a1bb65da45062=8fec6f603c4d482c32379d735fa77536; path=/; HttpOnly; Secure; SameSite=None
<
{ [4131 bytes data]
* Connection #0 to host console-openshift-console.apps.bcca4ad0ec252f076ae6.hypershift.aws-2.ci.openshift.org:443 left intact

The problem is that this doesn't work with browsers, which strictly require a successful ALPN negotiation during the TLS handshake to use HTTP/2 over HTTPS. So they are switching to HTTP/1.1.

I think it is worth mentioning ALPN here as this is central to the browser behaviour.

Claude review suggests that this distinction matters throughout the ADR:

  • Platform constraints: "HTTP/2 requires cluster-admin to enable at IngressController level" should say "h2 (HTTP/2 over TLS via ALPN) requires cluster-admin."
  • Ingress controllers: "HTTP/2 enabled by default with TLS" for Traefik/Caddy is imprecise. These support both h2 and h2c; the relevant point is that they negotiate h2 via ALPN automatically when TLS is configured.
  • The problem statement: The ADR should explicitly state that the bottleneck is about h2 specifically, because browsers only use h2. This helps readers understand why TLS and ALPN are central to the solution, and why h2c (which works today without cluster-admin) doesn't solve the browser performance problem.

Making this distinction explicit would help reviewers evaluate alternatives more clearly. For example, it immediately clarifies why "just enable HTTP/2 on the router" isn't as simple as it sounds, and why the sidecar needs to handle TLS.

@gazarenkov gazarenkov Jul 24, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"The problem is that this doesn't work with browsers" - exactly, so what do you propose? To add it as an alternative and reject because it does not work in browser?


## Decision

@rm3l rm3l Jul 21, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gazarenkov FYI (also shared during the standup call today), I raised the concerns here in the SOS call yesterday, and we agreed on the following next steps:

  • explicitly document the requirements for users to enable HTTP/2; so this won't be enabled by default, but users will be aware of what to do to improve performance if needed.
  • explicitly claim support for this setup, by essentially testing it on our supported platforms, as this was a concern raised by the customer in the support case

So I think we may no longer need this ADR for now, as we can "just" test and document the setup that was shared in the POC, so we can claim support for it, without having to update the install methods for now. I'll update the Epics under that Feature to reflect that.

@gazarenkov gazarenkov Jul 24, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ADR did its work - making it clear that initial requirement about HTTP/2 by default does not seem to be realistic and so we can fallback to the simplest possible HTTP/2 solution to try to workaround the frontend performance problem.
This is kinda cool and new option of ADR cycle.
IMO what we need to do is document here that requirements changed and => new decision made (does not matter if it is related to new functionality or "just" testing and documentation, it is still a decision). Current decision should be moved to another Alternative with corresponding rejection reason.


Introduce unified networking configuration for both OpenShift (Route) and vanilla Kubernetes (Ingress), with HTTP/2 proxy support:

- **OpenShift**: Provide optional sidecar proxy that handles TLS termination and HTTP/2, allowing users to enable HTTP/2 without cluster-admin privileges
- **Vanilla Kubernetes**: Add Ingress configuration; sidecar provides HTTP/2 without cluster-admin dependency (similar benefit as OpenShift)

**Implementation approach**:

1. **Add optional HTTP/2 proxy sidecar container** to Backstage deployment:
- Lightweight reverse proxy (e.g., NGINX, Envoy, Caddy) configured for HTTP/2 and TLS termination
- Proxies requests to Backstage on localhost:7007
- Mounts TLS certificates from Secret
- Can add caching headers for static assets (e.g., `/api/scalprum/*/static/*.js`) — reduces repeat downloads on subsequent page loads — see [POC](https://github.com/karthikjeeyar/rhdh-http2)

2. **Use passthrough termination** (when `http2Proxy` enabled):

**OpenShift Route:**
```yaml
apiVersion: route.openshift.io/v1
kind: Route
spec:
tls:
termination: passthrough

@rm3l rm3l Jul 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There might be understated consequences to switching from edge to passthrough termination. Could be stated under the consequences section?
From Claude, this is a significant operational change:

  • OpenShift no longer manages certificates for the route. With edge termination, the default wildcard cert or the route's configured certificate is managed by the router. With passthrough, the sidecar owns TLS entirely.
  • HAProxy-level HTTP headers are lost. The router no longer injects X-Forwarded-For, X-Forwarded-Proto, X-Forwarded-Port, etc. The sidecar must be configured to add these, or the Backstage application won't see the original client IP and protocol.
  • Monitoring and logging changes. HAProxy access logs and metrics no longer reflect individual HTTP requests for passthrough routes; they only see TCP connections.

@gazarenkov gazarenkov Jul 24, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's looks like truth :) so what must we do with this information?

port:
targetPort: https
```

**Kubernetes Ingress** (if http2Proxy used): Configure Ingress controller for SSL passthrough to let the sidecar handle HTTP/2 termination.

3. **Certificate provisioning options**:
- OpenShift service serving certificates (annotation-based, auto-rotated, but causes browser warnings — internal CA)
- cert-manager with Let's Encrypt (public CA, no browser warnings)
- User-provided certificates via Secret

4. **Expose via `spec.network`**:

**OpenShift (Route):**
```yaml
spec:
network:
route:
enabled: true
host: my-backstage.example.com
tls:
externalCertificateSecretName: my-tls
http2Proxy:
enabled: true # required for HTTP/2 on OpenShift without cluster-admin
```

**Vanilla Kubernetes (Ingress):**
```yaml
spec:
network:
ingress:
enabled: true
host: my-backstage.example.com
className: <ingress-class> # optional
tls:
secretName: my-tls
http2Proxy:
enabled: true # enables HTTP/2 without cluster-admin (NGINX Ingress also requires cluster-admin to enable HTTP/2)
```

- `spec.network.route` — moved from `spec.application.route` (deprecated, supported with warning)
- `spec.network.ingress` — new, for vanilla Kubernetes deployments
- `spec.network.http2Proxy.enabled` — adds sidecar; switches Route to passthrough / configures Ingress for SSL passthrough
- Certificate reuse: when `http2Proxy.enabled`, proxy uses `route.tls` or `ingress.tls` certificate; defaults to service serving certificates if not specified

**Note:** The `http2Proxy` sidecar provides HTTP/2 support without cluster-admin cooperation on both OpenShift and Kubernetes (where NGINX Ingress Controller also requires cluster-admin to enable HTTP/2).

5. **Default behavior considerations**:

@rm3l rm3l Jul 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be labeled explicitly as an "Open Question"?


Enabling `http2Proxy` by default is possible but has UX trade-offs:
- **Without user-provided certificate**: Uses OpenShift service serving certificates (internal CA) — HTTP/2 works but browsers show certificate warning

@rm3l rm3l Jul 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HTTP/2 works but browsers show certificate warning

Well, for clarity, HTTP/2 won't work until the user explicitly bypasses the certificate error in their browser, as ALPN negotiation happens during the TLS handshake itself.
As currently stated here, it sounds like a minor inconvenience, but in reality, without a user-provided certificate that matches the external RHDH hostname and is signed by a trusted CA (either internal or external to the company), the deployment is broken for browsers in any practical sense. I doubt any enterprise deployment would accept this IMO.

- **With user-provided certificate**: No warnings, full HTTP/2 benefits

@rm3l rm3l Jul 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should also clarify that the certificate should be trusted by browsers and match the RHDH hostname. Otherwise, they could provide a self-signed cert or one for the wrong hostname and would still get the same previous hard error in the browser, I guess.


**Options:**
- `http2Proxy.enabled: false` by default — users opt-in, no surprises
- `http2Proxy.enabled: true` by default — HTTP/2 out of the box, but certificate warning unless user provides `route.tls.externalCertificateSecretName`

If decided to default to `enabled: true`: provide clear documentation that users should supply their own certificate to avoid browser warnings. The performance benefit may justify the default, and the warning serves as a signal to configure properly.

@rm3l rm3l Jul 13, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If decided to default to enabled: true: provide clear documentation that users should supply their own certificate to avoid browser warnings. The performance benefit may justify the default, and the warning serves as a signal to configure properly.

The JIRA requirement is clear and this was my understanding as well: Enable HTTP/2 by default and Existing deployments gain HTTP/2 on upgrade without manual configuration changes. But I am concerned here about the TLS requirement. Defaulting to enabled would require users to provide certs and proper TLS configuration; this implies that the "upgrade without manual configuration" isn't really do-able here. We need to raise this.

@gazarenkov gazarenkov Jul 14, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, that's the point.
If we are not able to avoid manual configuration it seems to be nice but less important feature


## Alternatives Considered

@rm3l rm3l Jul 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks there is another possible alternative: Gateway API (GA in OCP and available in vanilla K8s). It seems to rely on Envoy on OCP and h2 works automatically via ALPN without any cluster-admin annotation. However, I've quickly checked on a ROSA 4.21 cluster that while the CRDs are installed, no GatewayClass is deployed by default; so a cluster-admin would still need to set up the Gateway infrastructure first. On vanilla K8s, Gateway API is a standard with multiple implementations but also requires installation. Worth mentioning as a forward-looking alternative with these caveats..

@rm3l rm3l Jul 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought there was an earlier revision of the ADR mentioning "Node.js native HTTP/2", or did I hallucinate? :)
Might be worth keeping it as alternative with clear reasoning why it is rejected, no?

@gazarenkov gazarenkov Jul 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can mention but I do not think it is about proxy so I did not yhink about it


### Alternative 1: Document cluster-admin HTTP/2 enablement
- **Approach**: Document how cluster-admins can enable HTTP/2 and require custom certificates per-route
- **Rejected because**: Users on shared clusters have no control; requires coordination with cluster-admin for each deployment; doesn't solve the core user autonomy problem

@rm3l rm3l Jul 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is true, but might also be the cleanest solution when cluster-admin cooperation is available: no sidecar, no TLS shift, no extra container..

@gazarenkov gazarenkov Jul 24, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might but does not fit initial requirements (no admin privileges :) )


### Alternative 2: Standalone HTTP/2 proxy Deployment
- **Approach**: Deploy NGINX/Envoy as separate Deployment + Service; Route/Ingress points to proxy, proxy routes to Backstage Service
- **Rejected because**: Single point of failure; extra network hop adds latency; doesn't scale automatically with Backstage replicas; more complex Service topology to manage

## Consequences

### Positive
✅ Sidecar approach enables HTTP/2 without cluster-admin privileges (bypasses Ingress controller limitations)
✅ Faster frontend page loads (multiplexing, reduced connections)
✅ Works on any OpenShift/Kubernetes cluster
✅ Lightweight proxy sidecars have minimal resource footprint
✅ Works correctly with multi-replica deployments (sidecar per pod)

### Negative
❌ HTTP/2 not available by default — requires user to provide certificate to avoid browser warnings
❌ Additional container per pod (slight resource overhead)
❌ Users must manage TLS certificates (unless accepting browser warnings with service serving certs)
❌ More complex deployment architecture to understand/debug
❌ Proxy configuration must be maintained by operator

## References

**Related issues:**
- [RHDHSUPP-320](https://redhat.atlassian.net/browse/RHDHSUPP-320)
- [RHDHPLAN-1598](https://redhat.atlassian.net/browse/RHDHPLAN-1598)
- [POC: rhdh-http2](https://github.com/karthikjeeyar/rhdh-http2)

**Documentation:**
- [FAQ: Does OpenShift Router support HTTP/2](https://docs.google.com/document/d/1S6E_sSmtxHknoVNBzW36q5Q-SHssbNFGWSWL3cwNEeo/edit?tab=t.0#heading=h.renxd4iwgobg)
- [Red Hat Blog: gRPC or HTTP/2 Ingress Connectivity in OpenShift](https://www.redhat.com/en/blog/grpc-or-http/2-ingress-connectivity-in-openshift)
Comment thread
gazarenkov marked this conversation as resolved.
- [OpenShift Docs: Configuring Routes](https://docs.redhat.com/en/documentation/openshift_container_platform/4.21/html/networking/configuring-routes)
- [NGINX Ingress Controller: ConfigMap HTTP/2 Setting](https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#listeners)
- [HTTP/2 connection coalescing explanation](https://daniel.haxx.se/blog/2016/08/18/http2-connection-coalescing/)
Comment thread
gazarenkov marked this conversation as resolved.
Loading