Skip to content

ci(e2e): use ephemeral host port for e2e Postgres to stop port-collision flake - #590

Open
yansun1996 wants to merge 2 commits into
ROCm:mainfrom
yansun1996:fix/e2e-postgres-ephemeral-port
Open

ci(e2e): use ephemeral host port for e2e Postgres to stop port-collision flake#590
yansun1996 wants to merge 2 commits into
ROCm:mainfrom
yansun1996:fix/e2e-postgres-ephemeral-port

Conversation

@yansun1996

Copy link
Copy Markdown
Member

What this fixes

The native-host E2E suite intermittently fails at accounting_cluster fixture setup with:

docker: Error response from daemon: failed to bind host port 0.0.0.0:55432/tcp: address already in use

This is a top recurring flake in the E2E / native-host job (e.g. run 31138116947 errored one accounting test and went green on re-run; run 31132066804's sibling run had a leaked container hold the port for ~3h, cascading the error across every accounting test).

Root cause

Each accounting test stands up its own Postgres container on node 0, but the fixture published them all to a fixed host port (-p 55432:5432). The ~30 accounting tests run serially, each start/stop cycling Postgres. docker rm -f returns before docker-proxy actually releases the host port, so the next test's docker run intermittently collides. A leaked container can hold the port for a whole run, turning a single flake into a full-suite cascade.

Approach

Stop pinning the host port. Publish with -p 5432 so Docker allocates a free ephemeral port atomically at bind time (race-free — no window between "check free" and "bind"), then discover the assigned port via docker port and thread it into database_url. A leaked/slow-release container now holds a random port, never the one the next test needs.

SPUR_TEST_PG_PORT still pins the port when explicitly set, preserving the deterministic-port escape hatch.

Testing

Ran on a real 2-node lab: 11 accounting tests, each cycling its own Postgres container serially (the exact pattern that produced the collision on the old code). All passed; containers came up on ephemeral host ports (32780, 32795, ...) and the harness resolved each correctly. A/B check confirmed the previously-observed pool timed out failure was an unrelated stale-process artifact on the shared lab node, identical with both fixed and ephemeral ports.

The accounting E2E fixture published every per-test Postgres container to
a fixed host port (55432). `docker rm -f` returns before docker-proxy
releases the host port, so the next test's `docker run` intermittently
failed with "address already in use", erroring the accounting_cluster
fixture at setup. A leaked container could hold the port for an entire
run, cascading the failure across all accounting tests.

Publish with `-p 5432` so Docker allocates a free ephemeral port
atomically at bind time, then resolve it via `docker port` and thread it
into database_url. SPUR_TEST_PG_PORT still pins the port when set.

Co-Authored-By: Claude <noreply@anthropic.com>
@yansun1996
yansun1996 requested a review from shiv-tyagi as a code owner August 7, 2026 06:29
Copilot AI lite review requested due to automatic review settings August 7, 2026 06:29
@yansun1996 yansun1996 changed the title fix(tests): use ephemeral host port for e2e Postgres to stop port-collision flake ci(e2e): use ephemeral host port for e2e Postgres to stop port-collision flake Aug 7, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a recurring native-host E2E flake where accounting tests intermittently fail to start Postgres due to collisions on a fixed host port, by switching to Docker-assigned ephemeral host ports and resolving the assigned port at runtime.

Changes:

  • Stop pinning Postgres to a fixed host port by default; allow Docker to allocate an ephemeral port (-p 5432).
  • Add logic to discover the ephemeral host port via docker port and construct database_url dynamically.
  • Preserve SPUR_TEST_PG_PORT as an escape hatch to explicitly pin the host port when needed.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/native_host/e2e/cluster.py
@codecov-commenter

codecov-commenter commented Aug 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #590      +/-   ##
==========================================
- Coverage   76.32%   76.28%   -0.04%     
==========================================
  Files         166      166              
  Lines       65511    65511              
==========================================
- Hits        49998    49973      -25     
- Misses      15513    15538      +25     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

docker run -p published the ephemeral-port Postgres container on all
interfaces with known test credentials, even though spurctld only ever
connects to it from the same node. Bind the publish spec to 127.0.0.1
explicitly; the existing docker port parsing is unaffected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants