[Egress] Observability: Metrics and Access Logging on Egress Gateway

[maksymvavilov]

Summary

Test that Istio proxy metrics are correctly emitted and queryable for egress traffic flowing through an Istio egress gateway. This validates that the egress gateway configuration (ServiceEntry, DestinationRule, HTTPRoute, Gateway) produces the expected observability signals as described in the Egress Observability Guide (kuadrant-operator#2108).

Egress observability differs from ingress in several key ways that the tests must account for:

• reporter is always "source" (no destination-side proxy on external services)
• destination_workload is "unknown"
• source_workload identifies the gateway, not the calling workload
• destination_service shows the external hostname from the ServiceEntry

Scope note: Access log tests are intentionally excluded. Access logs are generated entirely by Istio/Envoy — Kuadrant has no code that controls their format or content, and there are no unit tests for them in kuadrant-operator. Access logging is already globally enabled in the test cluster via meshConfig.accessLogFile in the Istio CR. The metrics tests below validate that the Kuadrant-configured egress topology produces correct Istio metric label values, which is the meaningful integration check.

The egress PodMonitor infrastructure introduced here is designed for reuse by follow-up egress observability scenarios: distributed tracing (kuadrant-operator#2077), TelemetryPolicy (kuadrant-operator#2081), and TokenRateLimitPolicy (kuadrant-operator#2079).

Setup

Builds on the existing egress fixture chain from testsuite/tests/singlecluster/egress/conftest.py (gateway, service_entry, destination_rule, hostname, client).

1. Prometheus — reuse the existing session-scoped prometheus fixture
2. PodMonitor — create a PodMonitor targeting the egress gateway pods (label selector: gateway.networking.k8s.io/gateway-name=kuadrant-egressgateway, metrics port: 15020, path: /stats/prometheus). Use the existing PodMonitor wrapper from testsuite/kubernetes/monitoring/
3. Egress gateway fixtures — reuse existing: Gateway, ServiceEntry, DestinationRule, HTTPRoute, httpbin backend, client

Test directory structure

testsuite/tests/singlecluster/egress/observability/
├── conftest.py              # PodMonitor fixture, egress metrics helpers
└── test_egress_metrics.py

Test Cases

Metrics Tests (test_egress_metrics.py)

Test 1: Egress request metrics have correct label values

Send a successful request through the egress gateway to the external httpbin service. Verify istio_requests_total is emitted with:

• destination_service = external hostname (e.g., httpbin.egress.local)
• source_workload = egress gateway workload name (e.g., kuadrant-egressgateway-istio)
• reporter = "source"
• response_code = "200"
• destination_workload = "unknown"

Use prometheus.wait_for_metric("istio_requests_total", ...) with label filters.

Test 2: Error responses are recorded in metrics

Send a request that triggers an error response from the external service (e.g., request to a path that returns 4xx/5xx). Verify istio_requests_total increments with the corresponding response_code label value.

Test 3: No-route requests produce NR response flag in metrics

Send a request to a hostname that has no matching HTTPRoute (e.g., unknown-api.example.com). Verify istio_requests_total is emitted with:

• response_flags = "NR"
• response_code = "404"
• destination_service = "unknown" (no ServiceEntry matched)

This validates that misconfigured clients are detectable via metrics.

Test 4: Latency histogram is populated

Send requests through the egress gateway. Verify istio_request_duration_milliseconds_bucket has entries with source_workload matching the egress gateway. This confirms latency data is available for PromQL percentile queries.

Dependencies

• Parent issue: testsuite#888 (Egress test epic)
• Egress gateway fixture chain from the basic egress test (testsuite#912)
• Prometheus fixture and monitoring CRD wrappers (existing)
• kuadrant-operator#2108 — upstream observability guide (reference)
• kuadrant-operator#2075 — parent task

References

• Egress Observability Guide (kuadrant-operator#2108)
• Egress Gateway Setup Guide
• Istio Standard Metrics
• Existing egress tests: testsuite#912 (AuthPolicy + RateLimitPolicy), testsuite#913 (Credential Injection)
• Follow-up scenarios: Distributed tracing (kuadrant-operator#2077), TelemetryPolicy (kuadrant-operator#2081), TRLP (kuadrant-operator#2079)