diff --git a/components/egress/docs/opentelemetry.md b/components/egress/docs/opentelemetry.md index 08b852fef..dd142f35c 100644 --- a/components/egress/docs/opentelemetry.md +++ b/components/egress/docs/opentelemetry.md @@ -14,8 +14,32 @@ This page lists the OpenTelemetry metrics currently implemented in egress. | `egress.policy.denied_total` | Counter | - | Number of DNS queries denied by policy. | | `egress.nftables.rules.count` | Observable Gauge | `{element}` | Approximate policy size after last successful static apply. | | `egress.nftables.updates.count` | Counter | - | Number of successful nftables updates (static apply + dynamic IP add). | -| `egress.system.memory.usage_bytes` | Observable Gauge | `By` | System memory used bytes (Linux: gopsutil; non-Linux build: `0`). | -| `egress.system.cpu.utilization` | Observable Gauge | `1` | CPU busy ratio in `[0,1]` (Linux: gopsutil; non-Linux build: `0`). | +| `egress.system.memory.usage_bytes` | Observable Gauge | `By` | **Node** memory used bytes (Linux: gopsutil; non-Linux build: `0`). | +| `egress.system.cpu.utilization` | Observable Gauge | `1` | **Node** CPU busy ratio in `[0,1]` (Linux: gopsutil; non-Linux build: `0`). | +| `egress.process.memory.usage_bytes` | Observable Gauge | `By` | Memory charged to the sidecar's own cgroup. Only present when cgroupfs is readable. | +| `egress.process.cpu.time` | Observable Counter | `s` | CPU seconds consumed by the sidecar's own cgroup. Only present when cgroupfs is readable. | + +### `system` vs `process` + +They measure different things, and the difference matters because this sidecar runs **per +sandbox**: + +- `egress.system.*` comes from gopsutil, i.e. `/proc/meminfo` and `/proc/stat`, which inside + a container describe the **node**. Every sandbox on a node therefore publishes the same + figure under its own `sandbox_id`. Do not chart these "by sandbox": the series look + per-sandbox but are N copies of one node number. Prefer kubelet/cAdvisor or a node + exporter for node-level data. +- `egress.process.*` is read from the sidecar's own cgroup (v2 `memory.current` and + `cpu.stat`, falling back to v1 `memory.usage_in_bytes` and `cpuacct.usage`), so it really + is per sandbox. + +`egress.process.cpu.time` is a **cumulative counter of consumed seconds**, not a sampled +ratio: use `rate()` on it. A ratio depends on the exporter's sampling interval, so it cannot +be re-aggregated or compared across differently configured deployments. + +Both `process` instruments are **registered only if their cgroup files can be read**. A +runtime that does not expose cgroupfs — a sandbox pod under `secure_runtime`, for instance — +gets no series at all, rather than a flat zero that reads like an idle sidecar. `egress.dns.query.duration` declares its bucket boundaries explicitly: @@ -46,7 +70,9 @@ resolutions with exhausted retry chains. All egress metrics may include shared attributes: -- `sandbox_id` from `OPENSANDBOX_EGRESS_SANDBOX_ID` (when set) +- `sandbox_id` from `OPENSANDBOX_EGRESS_SANDBOX_ID` (when set). Without it the sidecars of + different sandboxes export identical attribute sets, so their series collide in the + backend — which matters most for the per-sandbox `egress.process.*` gauges. - extra key/value attributes from `OPENSANDBOX_EGRESS_METRICS_EXTRA_ATTRS` (when set) ## OTEL Endpoint Configuration diff --git a/components/egress/pkg/telemetry/cgroup_linux.go b/components/egress/pkg/telemetry/cgroup_linux.go new file mode 100644 index 000000000..3a4e040c8 --- /dev/null +++ b/components/egress/pkg/telemetry/cgroup_linux.go @@ -0,0 +1,69 @@ +// Copyright 2026 Alibaba Group Holding Ltd. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//go:build linux + +package telemetry + +import ( + "os" + "path/filepath" +) + +// cgroupRoot is where the container's own cgroup is mounted. With a cgroup namespace — +// the default for containerd and CRI-O on cgroup v2 — this path is the container's cgroup +// root, so the values below describe the sidecar and not the node. Overridden in tests. +var cgroupRoot = "/sys/fs/cgroup" + +// processMemoryUsageBytes returns the sidecar's own memory usage, trying cgroup v2 before +// v1. The bool is false when neither layout is readable, which is a real possibility: a +// sandbox pod may run under a runtime that does not expose cgroupfs (see secure_runtime), +// and the caller must then skip the metric rather than publish a zero. +func processMemoryUsageBytes() (int64, bool) { + if data, err := os.ReadFile(filepath.Join(cgroupRoot, "memory.current")); err == nil { + if value, ok := cgroupSingleValueBytes(data); ok { + return value, true + } + } + // memory has its own v1 mount; unlike cpuacct it is not co-mounted with another + // controller in any layout worth supporting. + if data, err := os.ReadFile(filepath.Join(cgroupRoot, "memory", "memory.usage_in_bytes")); err == nil { + if value, ok := cgroupSingleValueBytes(data); ok { + return value, true + } + } + return 0, false +} + +// processCPUTimeSeconds returns the CPU time the sidecar has consumed, cgroup v2 first. +// Cumulative on purpose: a counter of consumed seconds composes with rate() and does not +// depend on the exporter's sampling interval, unlike a sampled utilisation ratio. +func processCPUTimeSeconds() (float64, bool) { + if data, err := os.ReadFile(filepath.Join(cgroupRoot, "cpu.stat")); err == nil { + if seconds, ok := cgroupCPUSecondsFromStat(data); ok { + return seconds, true + } + } + // cgroup v1: cpuacct may be mounted on its own or co-mounted with cpu, which is the + // systemd default and what a container usually inherits. Trying only the first layout + // would report CPU as unavailable on a readable cgroupfs. + for _, dir := range []string{"cpuacct", "cpu,cpuacct"} { + if data, err := os.ReadFile(filepath.Join(cgroupRoot, dir, "cpuacct.usage")); err == nil { + if seconds, ok := cgroupCPUSecondsFromNanos(data); ok { + return seconds, true + } + } + } + return 0, false +} diff --git a/components/egress/pkg/telemetry/cgroup_linux_test.go b/components/egress/pkg/telemetry/cgroup_linux_test.go new file mode 100644 index 000000000..7f8cf2786 --- /dev/null +++ b/components/egress/pkg/telemetry/cgroup_linux_test.go @@ -0,0 +1,133 @@ +// Copyright 2026 Alibaba Group Holding Ltd. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//go:build linux + +package telemetry + +import ( + "os" + "path/filepath" + "testing" +) + +// withCgroupRoot points the readers at a fake cgroupfs for the duration of a test. +func withCgroupRoot(t *testing.T, files map[string]string) string { + t.Helper() + root := t.TempDir() + for name, content := range files { + path := filepath.Join(root, name) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatalf("mkdir: %v", err) + } + if err := os.WriteFile(path, []byte(content), 0o644); err != nil { + t.Fatalf("write %s: %v", name, err) + } + } + previous := cgroupRoot + cgroupRoot = root + t.Cleanup(func() { cgroupRoot = previous }) + return root +} + +func TestProcessMetricsReadCgroupV2(t *testing.T) { + withCgroupRoot(t, map[string]string{ + "memory.current": "8036352\n", + "cpu.stat": "usage_usec 1500000\nuser_usec 1000000\n", + }) + + memory, ok := processMemoryUsageBytes() + if !ok || memory != 8036352 { + t.Errorf("memory = %d, %v; want 8036352, true", memory, ok) + } + seconds, ok := processCPUTimeSeconds() + if !ok || seconds != 1.5 { + t.Errorf("cpu = %v, %v; want 1.5, true", seconds, ok) + } +} + +func TestProcessMetricsFallBackToCgroupV1(t *testing.T) { + withCgroupRoot(t, map[string]string{ + "memory/memory.usage_in_bytes": "4096\n", + "cpuacct/cpuacct.usage": "2500000000\n", + }) + + memory, ok := processMemoryUsageBytes() + if !ok || memory != 4096 { + t.Errorf("memory = %d, %v; want 4096, true", memory, ok) + } + seconds, ok := processCPUTimeSeconds() + if !ok || seconds != 2.5 { + t.Errorf("cpu = %v, %v; want 2.5, true", seconds, ok) + } +} + +// The systemd default on cgroup v1 co-mounts cpu and cpuacct, and a container inherits +// that layout: cpuacct.usage then lives under cpu,cpuacct/ and not cpuacct/. +func TestProcessCPUReadsCoMountedCgroupV1(t *testing.T) { + withCgroupRoot(t, map[string]string{ + "cpu,cpuacct/cpuacct.usage": "3000000000\n", + }) + + seconds, ok := processCPUTimeSeconds() + if !ok || seconds != 3.0 { + t.Errorf("cpu = %v, %v; want 3, true", seconds, ok) + } +} + +// A runtime that does not expose cgroupfs must yield no reading at all. Publishing zero +// would be indistinguishable from an idle sidecar. +func TestProcessMetricsUnavailableWithoutCgroupfs(t *testing.T) { + withCgroupRoot(t, nil) + + if _, ok := processMemoryUsageBytes(); ok { + t.Error("memory reported available with no cgroup files") + } + if _, ok := processCPUTimeSeconds(); ok { + t.Error("cpu reported available with no cgroup files") + } +} + +// End to end: with a readable cgroupfs the instruments are registered and observed; with +// none, they must be absent from the collection rather than present and zero. +func TestProcessMetricsRegistrationFollowsAvailability(t *testing.T) { + t.Run("registered when readable", func(t *testing.T) { + withCgroupRoot(t, map[string]string{ + "memory.current": "8036352\n", + "cpu.stat": "usage_usec 1500000\n", + }) + + collected := collectEgressMetrics(t) + + if got := collected["egress.process.memory.usage_bytes"]; got != float64(8036352) { + t.Errorf("memory metric = %v, want 8036352", got) + } + if got := collected["egress.process.cpu.time"]; got != 1.5 { + t.Errorf("cpu metric = %v, want 1.5", got) + } + }) + + t.Run("absent when unreadable", func(t *testing.T) { + withCgroupRoot(t, nil) + + collected := collectEgressMetrics(t) + + if _, present := collected["egress.process.memory.usage_bytes"]; present { + t.Error("memory metric registered without a cgroup source") + } + if _, present := collected["egress.process.cpu.time"]; present { + t.Error("cpu metric registered without a cgroup source") + } + }) +} diff --git a/components/egress/pkg/telemetry/cgroup_other.go b/components/egress/pkg/telemetry/cgroup_other.go new file mode 100644 index 000000000..84f8ed59b --- /dev/null +++ b/components/egress/pkg/telemetry/cgroup_other.go @@ -0,0 +1,23 @@ +// Copyright 2026 Alibaba Group Holding Ltd. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//go:build !linux + +package telemetry + +// cgroups are Linux-only. Both report unavailable so the instruments are never +// registered, mirroring the hostmetrics_stub split. +func processMemoryUsageBytes() (int64, bool) { return 0, false } + +func processCPUTimeSeconds() (float64, bool) { return 0, false } diff --git a/components/egress/pkg/telemetry/cgroup_parse.go b/components/egress/pkg/telemetry/cgroup_parse.go new file mode 100644 index 000000000..de5d52b47 --- /dev/null +++ b/components/egress/pkg/telemetry/cgroup_parse.go @@ -0,0 +1,60 @@ +// Copyright 2026 Alibaba Group Holding Ltd. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package telemetry + +import ( + "strconv" + "strings" +) + +// cgroupSingleValueBytes reads a cgroup file holding one integer: memory.current on v2, +// memory.usage_in_bytes on v1. Returns false for "max" and for anything unparseable, so a +// caller never mistakes an unset limit or a garbled read for a measurement. +func cgroupSingleValueBytes(data []byte) (int64, bool) { + field := strings.TrimSpace(string(data)) + if field == "" || field == "max" { + return 0, false + } + value, err := strconv.ParseInt(field, 10, 64) + if err != nil || value < 0 { + return 0, false + } + return value, true +} + +// cgroupCPUSecondsFromStat pulls usage_usec out of a cgroup v2 cpu.stat. +func cgroupCPUSecondsFromStat(data []byte) (float64, bool) { + for _, line := range strings.Split(string(data), "\n") { + field, ok := strings.CutPrefix(strings.TrimSpace(line), "usage_usec ") + if !ok { + continue + } + micros, err := strconv.ParseInt(strings.TrimSpace(field), 10, 64) + if err != nil || micros < 0 { + return 0, false + } + return float64(micros) / 1e6, true + } + return 0, false +} + +// cgroupCPUSecondsFromNanos converts a cgroup v1 cpuacct.usage (nanoseconds) to seconds. +func cgroupCPUSecondsFromNanos(data []byte) (float64, bool) { + nanos, ok := cgroupSingleValueBytes(data) + if !ok { + return 0, false + } + return float64(nanos) / 1e9, true +} diff --git a/components/egress/pkg/telemetry/cgroup_parse_test.go b/components/egress/pkg/telemetry/cgroup_parse_test.go new file mode 100644 index 000000000..93dafd692 --- /dev/null +++ b/components/egress/pkg/telemetry/cgroup_parse_test.go @@ -0,0 +1,85 @@ +// Copyright 2026 Alibaba Group Holding Ltd. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package telemetry + +import "testing" + +func TestCgroupSingleValueBytes(t *testing.T) { + t.Parallel() + + cases := []struct { + name string + data string + want int64 + wantOK bool + }{ + {name: "memory.current", data: "8036352\n", want: 8036352, wantOK: true}, + {name: "no trailing newline", data: "4096", want: 4096, wantOK: true}, + {name: "zero is a real value", data: "0\n", want: 0, wantOK: true}, + // "max" means unlimited, not a measurement. + {name: "max", data: "max\n", wantOK: false}, + {name: "empty", data: "", wantOK: false}, + {name: "not a number", data: "eight\n", wantOK: false}, + {name: "negative", data: "-1\n", wantOK: false}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got, ok := cgroupSingleValueBytes([]byte(tc.data)) + if ok != tc.wantOK { + t.Fatalf("ok = %v, want %v", ok, tc.wantOK) + } + if ok && got != tc.want { + t.Errorf("value = %d, want %d", got, tc.want) + } + }) + } +} + +func TestCgroupCPUSecondsFromStat(t *testing.T) { + t.Parallel() + + // Real cpu.stat: usage_usec is not the first field, and more follow it. + stat := "usage_usec 1500000\nuser_usec 1000000\nsystem_usec 500000\nnr_periods 0\n" + got, ok := cgroupCPUSecondsFromStat([]byte(stat)) + if !ok { + t.Fatal("ok = false, want true") + } + if got != 1.5 { + t.Errorf("seconds = %v, want 1.5", got) + } + + // user_usec must not be mistaken for usage_usec. + if _, ok := cgroupCPUSecondsFromStat([]byte("user_usec 1000000\n")); ok { + t.Error("matched a field that is not usage_usec") + } + if _, ok := cgroupCPUSecondsFromStat([]byte("usage_usec notanumber\n")); ok { + t.Error("accepted an unparseable usage_usec") + } + if _, ok := cgroupCPUSecondsFromStat(nil); ok { + t.Error("accepted empty cpu.stat") + } +} + +func TestCgroupCPUSecondsFromNanos(t *testing.T) { + t.Parallel() + + got, ok := cgroupCPUSecondsFromNanos([]byte("2500000000\n")) + if !ok { + t.Fatal("ok = false, want true") + } + if got != 2.5 { + t.Errorf("seconds = %v, want 2.5", got) + } +} diff --git a/components/egress/pkg/telemetry/metrics.go b/components/egress/pkg/telemetry/metrics.go index 6405f3fb8..8dd9ea594 100644 --- a/components/egress/pkg/telemetry/metrics.go +++ b/components/egress/pkg/telemetry/metrics.go @@ -146,7 +146,57 @@ func registerEgressMetrics() error { return nil }), ) - return err + if err != nil { + return err + } + + return registerProcessMetrics() +} + +// registerProcessMetrics adds the sidecar's own resource usage, read from its cgroup. +// +// The egress.system.* gauges above come from gopsutil, i.e. /proc/meminfo and /proc/stat, +// which inside a container describe the node. Since this sidecar runs per sandbox, every +// sandbox on a node publishes the same node figure under its own sandbox_id — series that +// look per-sandbox but are not. The metrics here are the per-sandbox ones. +// +// Registration is conditional: if the cgroup files cannot be read the instruments are not +// created at all, so a missing source shows up as an absent series rather than a flat zero +// that reads like real data. +func registerProcessMetrics() error { + if _, ok := processMemoryUsageBytes(); ok { + if _, err := meter.Int64ObservableGauge( + "egress.process.memory.usage_bytes", + metric.WithDescription("Memory currently charged to the egress sidecar's own cgroup."), + metric.WithUnit("By"), + metric.WithInt64Callback(func(ctx context.Context, obs metric.Int64Observer) error { + if value, ok := processMemoryUsageBytes(); ok { + obs.Observe(value, egressMetricOpt()) + } + return nil + }), + ); err != nil { + return err + } + } + + if _, ok := processCPUTimeSeconds(); ok { + if _, err := meter.Float64ObservableCounter( + "egress.process.cpu.time", + metric.WithDescription("CPU seconds consumed by the egress sidecar's own cgroup."), + metric.WithUnit("s"), + metric.WithFloat64Callback(func(ctx context.Context, obs metric.Float64Observer) error { + if seconds, ok := processCPUTimeSeconds(); ok { + obs.Observe(seconds, egressMetricOpt()) + } + return nil + }), + ); err != nil { + return err + } + } + + return nil } func NftRuleCountFromPolicy(p *policy.NetworkPolicy) int64 { diff --git a/components/egress/pkg/telemetry/metrics_test.go b/components/egress/pkg/telemetry/metrics_test.go index e4e33bea7..33100e6ff 100644 --- a/components/egress/pkg/telemetry/metrics_test.go +++ b/components/egress/pkg/telemetry/metrics_test.go @@ -111,3 +111,44 @@ func dnsDurationDataPoint(t *testing.T, rm *metricdata.ResourceMetrics) metricda t.Fatal("egress.dns.query.duration not collected") return metricdata.HistogramDataPoint[float64]{} } + +// collectEgressMetrics registers the egress instruments against a fresh ManualReader and +// returns the single observed value per metric name. +func collectEgressMetrics(t *testing.T) map[string]float64 { + t.Helper() + + reader := sdkmetric.NewManualReader() + previous := otel.GetMeterProvider() + otel.SetMeterProvider(sdkmetric.NewMeterProvider(sdkmetric.WithReader(reader))) + t.Cleanup(func() { otel.SetMeterProvider(previous) }) + + if err := registerEgressMetrics(); err != nil { + t.Fatalf("registerEgressMetrics: %v", err) + } + + var rm metricdata.ResourceMetrics + if err := reader.Collect(context.Background(), &rm); err != nil { + t.Fatalf("collect: %v", err) + } + + out := map[string]float64{} + for _, sm := range rm.ScopeMetrics { + for _, m := range sm.Metrics { + switch data := m.Data.(type) { + case metricdata.Gauge[int64]: + for _, dp := range data.DataPoints { + out[m.Name] = float64(dp.Value) + } + case metricdata.Gauge[float64]: + for _, dp := range data.DataPoints { + out[m.Name] = dp.Value + } + case metricdata.Sum[float64]: + for _, dp := range data.DataPoints { + out[m.Name] = dp.Value + } + } + } + } + return out +} diff --git a/docs/components/egress.md b/docs/components/egress.md index bac58311d..1cecca9f0 100644 --- a/docs/components/egress.md +++ b/docs/components/egress.md @@ -204,6 +204,38 @@ millisecond ladder (`0, 5, 10, … 10000`), which would put every realistic DNS single `le=5` bucket and make `histogram_quantile()` return an interpolation rather than a measurement. +#### Resource usage: node vs sidecar + +Two pairs of gauges look interchangeable and are not: + +| Metric | Unit | Scope | +|---|---|---| +| `egress.system.memory.usage_bytes` | `By` | the **node** | +| `egress.system.cpu.utilization` | `1` | the **node** | +| `egress.process.memory.usage_bytes` | `By` | this **sidecar** | +| `egress.process.cpu.time` | `s` | this **sidecar** | + +The `system` pair comes from `/proc/meminfo` and `/proc/stat`, which inside a container +describe the node. Since the sidecar runs **per sandbox**, every sandbox on a node reports +the same node figure under its own `sandbox_id` — do not chart these "by sandbox", because +the series look per-sandbox and are N copies of one number. Use kubelet/cAdvisor or a node +exporter for node-level data. + +The `process` pair is read from the sidecar's own cgroup, so it really is per sandbox. +`egress.process.cpu.time` is a **cumulative counter of consumed seconds** — query it with +`rate()`. A sampled ratio would depend on the export interval and could not be compared +across deployments. + +Per-sandbox attribution needs `OPENSANDBOX_EGRESS_SANDBOX_ID` to be set, since that is what +becomes the `sandbox_id` attribute. Without it every sidecar exports the same attribute set +and the series from different sandboxes collide in the backend — which makes the `process` +metrics look flat or flapping rather than absent. Set it when launching the sidecar. + +Both `process` metrics are **only present when the sidecar's cgroup is readable** (cgroup v2 +`memory.current` / `cpu.stat`, or v1 `memory.usage_in_bytes` / `cpuacct.usage`). Under a +runtime that does not expose cgroupfs the series are absent rather than zero, so a flat zero +is never mistaken for an idle sidecar. + Full metric inventory and attribute semantics: [egress OpenTelemetry reference](https://github.com/opensandbox-group/OpenSandbox/blob/main/components/egress/docs/opentelemetry.md). ## Build & Run