Skip to content

refactor: shared build runner across platforms#2907

Draft
henryiii wants to merge 9 commits into
pypa:mainfrom
henryiii:refactor/shared-build-runner
Draft

refactor: shared build runner across platforms#2907
henryiii wants to merge 9 commits into
pypa:mainfrom
henryiii:refactor/shared-build-runner

Conversation

@henryiii

@henryiii henryiii commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Alternative to #2903, this time using Claude Fable 5 instead of Claude Opus 4.8. This is not just a one to one; this one doesn't have nearly as much restraining it, and I didn't tell it where I wanted to go later (--stage). But otherwise it's also an unedited output of a plan.

This is not mergable as it stands, but I think I can iterate on it to make something worth reviewing. I'll try to rebase it and work on it after 4.1.

🤖 AI text below 🤖

What

Extracts the per-wheel build loop that was duplicated across all six platform modules into a shared runner (cibuildwheel/platforms/runner.py), driven by a per-platform builder class with clearly-named steps:

setup → before_build → build_wheel → repair_wheel → audit_wheel
      → test_wheel → move_to_output → cleanup
  • Builder(Protocol[PathT]) — one method per step, checked structurally; there is no base class. Each platform's builder is a frozen dataclass whose fields are exactly the state its steps need. Generic over the wheel-path type because Linux wheels live in the container as PurePosixPath; everywhere else they're host Paths.
  • BuildSpec(identifier, setup) — a builder is only constructed once its environment is fully set up: setup() does the work and returns the ready builder, so every attribute is set for the object's whole lifetime.
  • run_builds() — a single flat loop that owns everything that was previously copy-pasted six times: log.build_start/build_end/step accounting, compatible-wheel (abi3) reuse, the none-any.whl check, exactly-one-repaired-wheel validation, AlreadyBuiltWheelError, the test gate, output bookkeeping, and cleanup. The loop body reads top-to-bottom like the old per-platform loops — there's no step registry or hook indirection.
  • host_*() functions — the step bodies shared by the five non-Linux platforms (before_build, default repair, audit, move-to-output, cleanup), taking a small HostBuilder protocol. Platforms delegate to them explicitly, so each platform module lists every step it performs.
  • run_before_all() / fatal_on_called_process_error() — replace the per-platform copies of the before_all hook and the CalledProcessError → FatalError wrapper (Linux keeps its per-container scope and troubleshoot() hook).

Each platform's build() shrinks to ~15 lines; platform-specific structure stays where it was and stays visible: macOS keeps its per-arch test loop (universal2/Rosetta), iOS its testbed flow, Android its dual build/android environments (the previous BuildState pattern was the prototype for this design), pyodide its pyodide venv testing, and Linux keeps get_build_steps() grouping and the OCIContainer lifecycle, invoking the runner once per container. build_in_container() keeps its signature (it's mocked in option_prepare_test.py), and the PlatformModule interface is unchanged.

Behavior changes

The duplication had let the platforms drift; this unifies the unjustified differences (one commit per platform, each itemized in its commit message):

Bugs fixed

  • pyodide: compatible-wheel reuse hit a latent NameError (assigned built_wheel, read repaired_wheel) — unreachable today only because pyodide wheels are never reused
  • pyodide: a repair command producing no wheel raised a raw StopIteration instead of RepairStepProducedNoWheelError
  • Linux: dependency-versions with packages: crashed because the host-side per-identifier temp dir was never created
  • Android: test-environment / CIBW_TEST_ENVIRONMENT_ANDROID was documented but silently ignored; it's now applied to before-test, the wheel install, and the testbed invocation

Unified behavior

  • repair must produce exactly one wheel on every platform (macOS/Windows/iOS gain RepairStepProducedMultipleWheelsError, previously Linux/Android-only)
  • Android gains the AlreadyBuiltWheelError duplicate-name check; its post-repair none-any.whl check is extended to every platform (previously the other five only checked the built wheel, so a repair command emitting a pure wheel slipped through)
  • Windows: the "ARM64 wheels cannot be tested" warning now only appears when a test command is actually configured
  • iOS: a test-suite failure raises FatalError instead of calling sys.exit(1) (same exit code)
  • pyodide: before_test no longer receives an undocumented {wheel} placeholder; test_command now receives {wheel} like Linux/macOS/Windows
  • AssertionError("uv not found")FatalError (macOS, Android)
  • cosmetics: one compatible-wheel reuse message everywhere, uniform log.step_end() accounting, host temp dirs removed with ignore_errors=True (pyodide previously never removed them), Android gains the "was moved to … instead of" warning

Testing

  • unit_test/runner_test.py (new): a recording FakeBuilder covers step ordering, compatible-wheel reuse (build/repair/audit/move skipped, test still runs), 0/2-wheel repair errors, duplicate-name and none-any errors, test gating, and the error wrapper + troubleshoot hook.
  • Existing suites pass unchanged: 834 unit tests (including the build() monkeypatch seams and option_prepare_test.py), doctests, ruff, strict mypy (3.11 + 3.14), pylint.
  • Integration run locally: pyodide (6 passed, end-to-end builds) and Linux in Docker (basic/before_all/testing/container selection). macOS CPython/Windows/iOS/Android rely on this PR's CI.

🤖 Generated with Claude Code

@joerick

joerick commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

I do like how the general run_builds function reads. And yes, centralisation of that logic is very sensible. Clever to side-step the Linux container complexity by just keeping that out of the refactor.

(The run_builds logic might actually be better as a method run() on the Runner/Builder, since the plurality doesn't do much.)

Downsides in my head:

  • one now has to mentally keep hold of run_builds and HostBuilder when reading any platform code to understand the context in which it's running
  • there's quite a lot of 'globalish' state on the runners (all the variables set on self). it's kinda hard to say anything about when/how that state is set. I guess most of it is set in __init__() and setup()? Perhaps the builder object could only be created once all the setup() code has completed? That way we don't have ambiguity around what variables are/arent set?

henryiii added 7 commits July 9, 2026 01:41
Adds cibuildwheel/platforms/runner.py: a Builder ABC (generic over the
wheel path type, since Linux wheels live in the container as
PurePosixPath) whose methods are the platform-specific build steps, and
run_builds(), a single flat loop that owns everything identical across
platforms: logging, compatible-wheel reuse, built/repaired wheel
validation, the test gate, output bookkeeping, and cleanup.

HostBuilder provides the step bodies shared by the five non-Linux
platforms. run_before_all() and fatal_on_called_process_error() replace
the per-platform copies of the before_all hook and the
CalledProcessError-to-FatalError wrapper.

No platform uses this yet; subsequent commits port them one at a time.

Assisted-by: ClaudeCode:claude-fable-5
The per-wheel loop in build() is replaced by a PyodideBuilder driven by
runner.run_builds(). Behavior changes:

- a compatible-wheel (abi3) reuse no longer crashes with a NameError
  (the old loop assigned built_wheel but read repaired_wheel; it was
  unreachable in practice since pyodide wheels are never reused)
- a repair command producing zero wheels now raises
  RepairStepProducedNoWheelError instead of a raw StopIteration, and
  producing several is now an error, like on Linux
- before_test no longer receives the undocumented {wheel} placeholder;
  test_command now does, matching the other platforms
- the per-identifier temp dir is now removed after each build

Assisted-by: ClaudeCode:claude-fable-5
The per-wheel loop becomes a MacOSBuilder; the per-architecture test
loop (universal2 testing both halves, Rosetta emulation, arch-specific
test-skip identifiers) stays inside test_wheel(), unchanged.

Behavior changes:
- a repair command producing several wheels is now an error, as on Linux
- a missing uv now raises FatalError instead of AssertionError
- the per-identifier temp dir is removed with ignore_errors, like Windows

Assisted-by: ClaudeCode:claude-fable-5
The per-wheel loop becomes a WindowsBuilder. The ARM64-on-non-ARM64
test skip moves inside test_wheel(), so its warning now only appears
when a test command is actually configured. A repair command producing
several wheels is now an error, as on Linux.

Assisted-by: ClaudeCode:claude-fable-5
The per-wheel loop becomes an IOSBuilder; the testbed-based test flow,
the non-simulator/non-native skip steps, and the python-module test
command validation stay inside test_wheel().

Behavior changes:
- a test-suite failure now raises FatalError instead of calling
  sys.exit(1) directly (same exit code, consistent error reporting)
- a repair command producing several wheels is now an error, as on Linux
- the compatible-wheel reuse message now matches the other platforms
- the per-identifier temp dir is removed with ignore_errors

Assisted-by: ClaudeCode:claude-fable-5
The BuildState dataclass and its step functions fold into an
AndroidBuilder; setup_target_python/setup_env and the other
environment helpers are unchanged.

Behavior changes:
- test-environment (CIBW_TEST_ENVIRONMENT_ANDROID) is now applied to
  the test steps (before-test, wheel install, testbed invocation); it
  was documented but previously ignored on Android
- a wheel name colliding with an earlier build now raises
  AlreadyBuiltWheelError, like every other platform
- the redundant post-repair none-any.whl check is dropped (the built
  wheel is still checked, like every other platform)
- the compatible-wheel reuse message now matches the other platforms,
  and the moved-elsewhere warning from move_file is now emitted
- a missing uv now raises FatalError instead of AssertionError

Assisted-by: ClaudeCode:claude-fable-5
The per-config loop in build_in_container() becomes a LinuxBuilder
(generic over PurePosixPath — wheels stay container-side, and the
existing copy-out and host-side audit behavior is preserved as
overridden steps). build() keeps the container grouping, engine check,
and OCIContainer lifecycle; the CalledProcessError wrapper keeps its
per-container scope and troubleshoot() hook via
fatal_on_called_process_error. build_in_container() keeps its
signature, since tests mock it.

Also fixes a latent bug: the host-side per-identifier temp dir was
never created, so dependency-versions with packages: would have
crashed writing constraints.txt on Linux.

Assisted-by: ClaudeCode:claude-fable-5
@henryiii henryiii force-pushed the refactor/shared-build-runner branch from 39fdbba to db751f8 Compare July 9, 2026 05:44
Rework the builder design in response to review feedback:

- Builder is now a Protocol rather than an ABC; platform builders are
  plain frozen dataclasses with no base class, checked structurally.
- A builder is only constructed once its setup has completed.
  run_builds() takes BuildSpec(identifier, setup) pairs, where setup()
  prepares the environment and returns the ready-to-use builder, so
  every attribute on a builder is set from the moment it exists and the
  dataclass fields document exactly what state each platform carries.
- The step bodies previously inherited from the HostBuilder base class
  are now module functions (runner.host_before_build() etc.) taking a
  small HostBuilder protocol; platforms delegate to them explicitly, so
  each platform file lists every step it performs. The two functions
  that run user commands take env explicitly, which also makes
  Android's dual-environment usage visible at the call sites.

Assisted-by: ClaudeCode:claude-fable-5
@henryiii

henryiii commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

I still need to review and see how well it did.

🤖 AI text below 🤖

Both downsides felt real, and @henryiii also wanted to move away from inheritance, so the latest commit reworks the design:

  • State ambiguity: fixed the way you suggested — a builder is now only constructed once all setup has completed. run_builds() takes BuildSpec(identifier, setup) pairs; setup() does the environment work and returns the ready builder. Builders are frozen dataclasses, so their fields document exactly what state each platform carries, every field is set for the object's whole lifetime, and mypy verifies the fields against the protocol.
  • Mental context: Builder is now a Protocol and the former HostBuilder base class is gone. The shared step bodies are plain functions (runner.host_repair_wheel(...) etc.) that platforms call explicitly, so each platform module now lists every step it performs — nothing arrives invisibly from a base class. You still need run_builds() for the driving order, but that's one function with the order in its docstring.

A small bonus: host_before_build()/host_repair_wheel() take env explicitly, which makes Android's dual build/android environment usage visible at the call sites instead of being implied by an inherited attribute.

On run() vs run_builds(): the plurality earns its keep through built_wheels — compatible-wheel (abi3) reuse spans identifiers, so the loop owns that list. Happy to revisit if you'd rather see it shaped differently.

Android checked the repaired wheel for none-any and its test suite
asserts that; the other platforms only checked the built wheel, so a
repair command emitting a pure wheel slipped through silently. Rather
than dropping Android's check, extend it: run_builds() now validates
the repaired wheel too, on all platforms.

Assisted-by: ClaudeCode:claude-fable-5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants