ci: build every engine arch from one gclient sync (single job, serial) - #22
Draft
jwinarske wants to merge 6 commits into
Draft
ci: build every engine arch from one gclient sync (single job, serial)#22jwinarske wants to merge 6 commits into
jwinarske wants to merge 6 commits into
Conversation
…e.sh
The four per-arch workflows each inlined the same depot_tools/gclient/gn/ninja/
prepare-sdk orchestration, differing only by arch, triple, and sysroot. Replace
them with a single matrix workflow that drives scripts/build-engine.sh — the
shared recipe (also consumed by emb and, next, a meta-flutter Yocto recipe), so
the orchestration lives in exactly one place.
- Add scripts/build-engine.sh: fetch (gclient sync) + build (patch -> gn ->
ninja -> prepare-sdk) phases; derives linux_cpu/triple/sysroot from the arch,
applies the per-arch patch series, and stages engine-sdk/. Validated locally
producing byte-comparable engines for x86_64/arm64/riscv64/armv7hf.
- Add patches/series-{default,riscv64}: the existing per-arch patch sets, with a
strip level so the recipe applies the same patches git apply reads today (no
patch files changed).
- flutter-engine.yaml matrix builds all arches x their modes (arm64/armv7hf/
x86_64: debug/debug-unopt/profile/release; riscv64: debug/profile/release),
preserving the exact linux-engine-sdk-<mode>-<arch>-<srcrev> tag names and the
draft/upload release flow.
Signed-off-by: Joel Winarske <joel.winarske@linux.com>
workflow_dispatch only works once a workflow is on the default branch, so add a branch-scoped push trigger (default srcrev, release=false) to exercise this on the adoption branch. Revert this commit before merging. Signed-off-by: Joel Winarske <joel.winarske@linux.com>
Rework the engine workflow from a four-way per-arch matrix — each job doing its own full gclient sync — into one job that fetches the source closure ONCE and builds every (arch, runtime-mode) from it. The closure (flutter monorepo + DEPS) is arch-independent, so it is synced once; each arch adds only its Debian sysroot (installed side by side under build/linux/) and its own gn invocation + out dir. That removes three redundant multi-tens-of-GB syncs per run. The patch series is applied once to the shared tree. For that to be safe the superset (patches/series-all) must coexist across arches: 0003 only adds riscv as a valid --linux-cpu, 0005/0007 are arch-neutral source fixes, and swiftshader-0002 is made arch-conditional (GN if/else on current_cpu) so riscv64 gets llvm-16 while every other arch keeps the llvm-10 Reactor it was validated against. Both llvm trees are already vendored, so this changes no non-riscv build. Builds run serially: one ninja -j nproc already saturates the cores, so parallel arches/modes add no throughput, only memory pressure (overlapping LTO release links exhaust RAM). Each packaged combo's out dir is pruned to bound disk, since all combos now share one checkout. Per-combo log groups keep progress legible. Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
jwinarske
changed the title
The self-hosted runner shares the build machine with interactive work; a full -j nproc sync/ninja starves it. Set EMB_JOBS=nproc-1 (min 1) in the job env so both the gclient sync and every ninja build leave one core free. Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
The release-mode LTO link is a single-threaded RAM spike that EMB_JOBS cannot bound, and the self-hosted runner shares the machine with interactive work. Set EMB_NO_LTO=1 for non-release runs (push + release=false dispatch); release=true keeps LTO so published engines stay production builds. The release input is read via an env var, not inlined, to avoid run-step injection. Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
Empty commit to exercise the reworked workflow on the self-hosted runner via the temporary push trigger. Drop before merge along with the push trigger. Signed-off-by: Joel Winarske <joel.winarske@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the four per-arch engine workflows with one job that syncs the engine
source once and builds every
(arch, runtime-mode)from that single closure,driving the shared
scripts/build-engine.shrecipe (the same recipe emb and theYocto layer call).
Why
The old matrix ran four jobs, each doing its own full
gclient sync— thesame arch-independent flutter monorepo + DEPS, tens of GB, fetched four times.
The closure does not depend on the target arch; only the sysroot, gn triple, and
out dir do. So: sync once, add each arch's Debian sysroot side by side under
build/linux/, build each target into its own out dir.Single shared patched tree
The patch series is applied once to the shared tree, so the superset must
coexist across arches (
patches/series-all):0003-gn-riscvonly adds riscv as a valid--linux-cpu— inert for others.0005-fml/0007-atkare arch-neutral source fixes (0007 is a no-op on theolder glib in the CI sysroots).
swiftshader-0002is now arch-conditional (GNif/elseoncurrent_cpu):riscv64's Reactor gets
llvm-16, every other arch keeps thellvm-10it wasvalidated against. Previously an unconditional
llvm-10 -> llvm-16bump, whichwould have silently switched arm64 (it also uses the LLVM Reactor —
supports_subzerois false for arm64 and riscv64) onto an unvalidated llvm-16.Both llvm trees are already vendored, so no non-riscv build changes.
Serial, not parallel
One
ninja -j nprocalready saturates the cores, so running arches/modesconcurrently adds no throughput — only memory pressure (overlapping LTO release
links exhaust RAM). Builds run serially; each packaged combo's out dir is pruned
to bound disk (all combos share one checkout); per-combo
::group::log sectionskeep progress legible.
Before merge
push:trigger (kept so the workflow can be exercisedfrom this branch;
workflow_dispatchonly works once it is on the defaultbranch).
went green under the previous matrix form; the recipe itself is unchanged).
Note: the self-hosted runner shares the build machine with graphics testing;
the "Flutter Engine" workflow is currently disabled while that machine is being
looked at, so validation is deferred.