test(e2e): wait for etcd quorum convergence in k0s multi-cp test

[yansun1996]

What this fixes

The native-k0s multi-control-plane E2E test test_three_control_planes_form_etcd_quorum is intermittently red in the E2E / native-host job:

assert 2 == 3   where 2 = etcd_member_count()

The surrounding CI runs pass, so it's a flaky convergence race, not a regression.

Root cause

The cluster's phase: ready is set as soon as every k0scontroller unit is systemd-active (converge_provisioning gates on each node's component state reported as active, which is the unit's systemctl ActiveState). Systemd-active precedes the embedded etcd finishing quorum formation: a secondary control-plane's unit is active the moment the k0s process starts, but its etcd member joins the quorum a short moment later.

The test asserted etcd_member_count() == 3 in a single shot immediately after wait_k8s_phase("ready"), with no settle window, so it could observe etcd mid-join (2 of 3). The failing run's own output confirms this: phase ready, all three control planes assigned, all three controllers active, but etcd had only 2 members at that instant.

This is a test-only timing gap. Marking the cluster ready on unit-active (then self-healing) is reasonable product behavior; etcd quorum is eventually-consistent behind it. Only the test treated the two as synchronous.

Approach

Add a wait_etcd_members(count, timeout=120) helper to the harness that polls etcd_member_count() until it reaches the target, mirroring the existing wait_k8s_phase poller, and use it in place of the one-shot read.

The test keeps its diagnostic value: a permanent stall below the target returns the last observed count, so assert members == 3 still fails (with the observed count in the message), and an over-count also fails rather than being masked.

How it was tested

• The helper is pure poll logic; its behavior was verified deterministically against the real function (patching only the member-count source and the clock): converges and early-returns on 1→2→3, runs to timeout and returns 2 on a permanent stall (assertion would fail), ignores transient zero-count RPC blips without short-circuiting, and surfaces an over-count.
• The full 3-node etcd path needs a 3-control-plane cluster, which CI provides; that end-to-end path was not run locally.

[Copilot]

Pull request overview

This PR reduces flakiness in the native-host k0s multi-control-plane E2E suite by adding an explicit wait for embedded etcd quorum formation after the cluster reports phase: ready.

Changes:

• Add a wait_etcd_members(count, timeout=120) polling helper to the E2E cluster harness.
• Update the test_three_control_planes_form_etcd_quorum test to wait for etcd membership convergence before asserting the quorum size.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
tests/native_host/e2e/test_k8s_multicp.py	Switch etcd quorum assertion to a polling-based wait to avoid mid-join races.
tests/native_host/e2e/cluster.py	Introduce wait_etcd_members helper to poll etcd_member_count() until the expected member count is observed or timeout.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #591      +/-   ##
==========================================
+ Coverage   76.30%   76.32%   +0.02%     
==========================================
  Files         166      166              
  Lines       65511    65511              
==========================================
+ Hits        49984    49999      +15     
+ Misses      15527    15512      -15     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.