Zenithar/chaos controller/full bpf network filter#1057
Draft
Zenithar wants to merge 19 commits into
Draft
Conversation
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 4ac738b | Docs | Datadog PR Page | Give us feedback! |
83daa37 to
fd5001a
Compare
…r network disruption - vendor testcontainers-go v0.42.0 + 8 transitive deps (all MIT/Apache/BSD) - add `make test-integration` target: cross-compile for Linux, build Docker image, run privileged with /proc + docker.sock mounted - add Dockerfile.integration (debian:12-slim + iproute2 + iptables + wget) - add integration suite scaffold (injector/integration_suite_test.go) with //go:build integration tag and env guards for CHAOS_INJECTOR_MOUNT_PROC - add SPEC.md, tasks/plan.md, tasks/todo.md, docs/ideas/ Suite design: TestIntegration shares the Ginkgo tree with existing unit tests (158 unit specs pass inside Docker); integration specs will be added on top. Env guards skip the suite cleanly when not running via make test-integration. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…tency structural)
T5: test helpers
- buildNetworkInjector: real tc/iptables/netlink/netns, mock cgroup (v2 noop),
fake k8s, mock BPF CmdRunner; returns (Injector, targetPID uint32)
- startIsolatedNetwork/startTargetContainer/startSenderContainer via testcontainers
- assertTCQdisc/assertTCQdiscAbsent: nsenter from test container into target netns
(avoids docker exec tc which requires tc in target image)
- Switch from -v /proc:/mnt/proc to --pid=host; CHAOS_INJECTOR_MOUNT_PROC=/proc/
- Multi-stage Dockerfile: compile BPF object in builder stage (engine.Attach requires
bpf-network-disruption.bpf.o even for delay-only specs)
T6: first vertical slice GREEN
- 200ms latency disruption: Inject → assertTCQdisc("delay 200ms") → Clean → no netem
- 1 Passed, 0 Failed, 0 leaked containers, 0.63s
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…et loss, clean-state)
Phase 2 scenarios — all 7 specs green, 56s wall-clock, 0 leaked containers.
New helpers:
- measureHTTPLatencyMS: curl -w %{time_total} inside sender container (correct ms timing;
`date +%s%3N` is unusable on Alpine/busybox)
- measurePingLoss: iputils ping inside sender, parses "X% packet loss"
- injectAndActivate: Inject() + tc matchall filter on target's eth0/lo to route all
egress traffic through netem band (1:4)
Architecture note: BPFDisruptCmdRunner remains mocked because the bpf-network-disruption
CGO binary (libbpfgo) cannot be built against Debian 12's libbpf 1.1.2 (requires 1.2.x).
The tc matchall filter is equivalent to the BPF match-all rule for empty-Hosts specs,
making behavioral assertions valid for the test scenarios.
Behavioral thresholds (conservative):
- Latency: baseline <100ms, disrupted >150ms (200ms injected, testcontainers exec overhead)
- Packet loss: baseline 0%, disrupted >20% (50% drop, TCP retransmits reduce observable loss)
- Clean-state: latency <100ms after Clean()
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
- fix unconvert lint error in network_disruption.go (unnecessary interface cast) - switch Dockerfile.integration base from debian:12 to ubuntu:24.04 - document make test-integration in docs/development.md (prerequisites, what it tests) - license check passes (new transitive deps: MIT/Apache/BSD, no GPL) Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…vioral tests Three root-cause bugs fixed: 1. BPF map name truncation: disruption_rules (16 chars) exceeded the 15-char BPF_OBJ_NAME_LEN limit. Kernel stored truncated name; binary searched full name → 0 maps found → LPM trie never populated. Fix: rename to disruption_rl. 2. cls_bpf return value: original returned -1 (TC_ACT_UNSPEC) for match. In Linux 5.x+ this overwrites classid with 0xFFFFFFFF (invalid class). Fix: return TC_CLASSID_DISRUPTION_BAND = TC_H_MAKE(1,4) = 0x00010004. 3. skb->data layout: in tc egress BPF on veth/bridge, skb->data starts at L3 (IP header), not L2 (Ethernet). Original EtherType check at data+12 read TTL/protocol bytes, never matched. Fix: dual L2/L3 detection with proper BPF verifier bounds checks. Integration test: injectAndActivate() uses tc matchall as safety net alongside the BPF classifier. binary returns error when 0 disruption_rl maps found. Dockerfile: ubuntu:24.04 multi-stage builds binary+object from source. 7/7 integration tests pass, 57s wall-clock. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…on flag
## BPF engine
- Add `direction` field (byte 17) to `lpm_val` struct so egress and ingress
BPF hooks can each skip rules that don't apply to their direction.
- Egress hook: perform LPM lookup; skip rules with `DIR_INGRESS`; fall back to
disruption band when no rule matches (preserves match-all behaviour).
- Ingress hook: skip rules with `DIR_EGRESS`.
- Add `IPPROTO_ICMP` (1) and `IPPROTO_ICMPV6` (58); `match_l4` returns true
immediately for ICMP/ICMPv6 matches (no port check for those protocols).
- Add `DirBoth = 0` constant; ALLOW rules (safeguards + allowedHosts) use
`DirBoth` so both hooks honour them — fixes a silent semantic bug where
`DirEgress` ALLOW rules were skipped by the ingress hook.
## CLI (`bpf-network-disruption`)
- Add `--direction egress|ingress|both` flag; `populateRules` passes it for
every rule.
- Add `icmp` and `icmpv6` to `--protocol`; `parseProtocol` cases sorted
alphabetically to match API enum order.
## API
- Extend `protocol` enum to `icmp|icmpv6|tcp|udp` on `NetworkDisruptionHostSpec`
and `NetworkDisruptionServiceSpec`.
## Integration tests
- New `network_disruption_ingress_integration_test.go`: 4 tests covering
ingress latency (IFB structural + behavioral) and ingress drop (BPF
structural + behavioral).
- New helpers: `ingressLatencySpec`, `ingressDropSpec`, `tcQdiscShowDev`,
`listNetDevices`, `tcFilterShowIngress`, `injectAndActivateIngress`,
`injectAndActivateIngressDrop`.
- Wire `BPFConfigInformerMock` in `buildNetworkInjector` — removes bpftool
runtime dependency for all integration tests.
- `tcQdiscShow` delegates to `tcQdiscShowDev("eth0")` — eliminates duplication.
## Infra
- `Dockerfile.integration`: install `linux-tools-generic` + bpftool symlink
fallback; verify `bpftool version` during build.
- `Makefile test-integration`: `docker build -q`, tee full log to
`/tmp/chaos-integration-test.log`, filter DEBUG lines from terminal output.
## Docs & examples
- Update protocol enum docs across `disruption_catalogue.md`, `flow.md`,
`hosts-and-services.md`, `cloud-managed-services.md`, `safemode.md`,
`network_disruption.md`.
- Remove stale "ingress only works for TCP" constraint.
- Expand `examples/network_ingress.yaml` with TCP drop, delay, and ICMP
variants; add new `examples/network_icmp.yaml`.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
fd5001a to
b2d401c
Compare
Aligns with disk-failure and network-tc-filter: the !cgo tag excludes the file from go vet ./... in environments where libbpf-dev is absent (CI). The ebpf/Makefile compiles via `go build main.go` (file-level), which bypasses build constraints, so the binary still builds correctly in the Docker build environment where libbpf headers are present. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
make manifests adds icmp and icmpv6 to the protocol field enum in all three Disruption CRD schemas (Disruption, DisruptionCron, DisruptionRollout). Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…protocol Protocol enum now includes icmp and icmpv6 so the validation error message changes from "udp, tcp" / "tcp, udp" to "icmp, icmpv6, udp, tcp" / "icmp, icmpv6, tcp, udp". Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…ry math
Three compounding issues caused the e2e job to be killed by the 45min
GitHub Actions timeout (appearing to "hang"):
1. Outer bash retry loop (3×) × Ginkgo --timeout=25m = 75min > 45min.
Ginkgo already has --flake-attempts=3; the outer loop was redundant.
Remove it.
2. Four ingress-shaping BPF tests (Delay/BandwidthLimit/Corrupt + ingress
flow) consistently failed in DryRun mode because:
- NewNetworkDisruptionInjector called ebpf.NewConfigInformer() even in
DryRun, which ran `bpftool -j feature probe` unconditionally.
- ValidateNetworkDisruptionConfig() then checked /boot/config-<kernel>,
absent in minikube Docker containers → "kernel config file not found".
- Injector exited without writing readiness probe → disruption never
reached Injected status → ExpectDisruptionStatus blocked ~5m37s each.
- 4 tests × 3 flake-attempts × ~5m37s/4 procs ≈ exceeded 25min Ginkgo
timeout.
Fix: skip NewConfigInformer and BPF validation entirely in DryRun mode.
Also fix defaultBpftoolExecutor.Run() to respect the dryRun flag (was
silently ignored unlike every other executor in the codebase).
3. CreateRunningPod condition: `len(runningPods) == 0 || allContainersAreRunning`
returned nil immediately when the pod was still Pending (wrong operator).
Fix: `len(runningPods) > 0 && allContainersAreRunning`.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Replaces the legacy iptables/u32-based network disruption packet filtering with a full eBPF (TC classifier) implementation for both ingress and egress traffic paths. Subsequent commits harden the engine, add a Docker-based integration test harness, and extend the BPF data plane with direction-aware rules, ICMP/ICMPv6 protocol support, and ingress-path behavioral tests.
Key changes
BPF data plane
bpfdisruptpackage —Enginemanages the BPF data plane: clsact qdisc lifecycle, IFB device for ingress shaping redirect, BPF program attachment (TC egress classifier + ingress DirectAction), and LPM trie map population for target CIDR matching.lpm_valcarries adirectionfield (DirBoth=0,DirEgress=1,DirIngress=2). The egress BPF hook skipsDIR_INGRESSrules; the ingress hook skipsDIR_EGRESSrules. ALLOW/safeguard rules useDirBothso both hooks honour them.IPPROTO_ICMP(1) andIPPROTO_ICMPV6(58) added tomatch_l4; port matching is skipped for those protocols.bpf-network-disruptionCLI — new--direction egress|ingress|bothand--protocol icmp|icmpv6|tcp|udpflags;populateRulespasses--directionfor every rule.Injector
flow: ingresswith delay/bandwidth/corrupt/duplicate, an IFB (Intermediate Functional Block) virtual device is created; matched ingress packets are redirected viabpf_redirectand shaped by a netem/tbf chain on the IFB.ActionDroprules apply probabilistic drop directly in the BPF ingress hook (TC_ACT_SHOT) without an IFB device.DirBoth— safeguard IPs (node IP, default gateway) andallowedHostsentries are now direction-neutral so they are honoured on both the egress and ingress hooks.Integration tests
Dockerfile.integration,make test-integration) with isolated per-test bridge networks, realtc/iptables/netlink drivers, andtestcontainers-gocontainers.BPFConfigInformerMockwired intobuildNetworkInjector— removes bpftool runtime dependency.make test-integrationnow tees full output to/tmp/chaos-integration-test.logand filters DEBUG lines from the terminal.API
protocolenum extended toicmp | icmpv6 | tcp | udponNetworkDisruptionHostSpecandNetworkDisruptionServiceSpec.Docs & examples
examples/network_ingress.yaml; addedexamples/network_icmp.yaml.lpm_valgains adirectionbyte at offset 17 (was padding). Struct size unchanged (20 bytes).disruption_rlBPF map directly (e.g.bpftool map dump) will see the new field. In-flight disruptions during a rolling upgrade will have stale map entries without the direction bit set — they default toDirBoth=0(match both directions), which is safe.bpf-network-disruptionCLI--directionis now passed for every rule. Old injector pods (pre-upgrade) do not pass--direction; the binary defaults to 0 (DirBoth), which preserves previous behaviour. New binary, old engine:--directionflag unrecognised — upgrade engine and binary together.Direction.String()outputDirBoth.String()now returns"both"(was"unknown"for zero value). Any code that compared against"unknown"for the zero direction will no longer match.protocolvalidationicmpandicmpv6are now valid values. Existing disruptions withprotocol: tcporprotocol: udpare unaffected.allowedHostsrules previously stored withDirEgressare now stored withDirBoth. In-flight disruptions during upgrade: old rules in the LPM map haveDirEgress; the new ingress hook skips them (returnsTC_ACT_OK), so safeguarded hosts continue to pass ingress traffic — behaviour is preserved but for the wrong reason until the disruption is re-injected.DirBothALLOW rules.flow: ingress) still create a netem qdisc on the egress side ofeth0becauseapplyOperationsToIfacesalways runs. Egress traffic is shaped by that netem. This pre-existing limitation is unchanged.Code Quality Checklist
Testing
unittests andintegrationtests (Docker-based,make test-integration).