[lstein]

Summary

Moved from invoke-ai#261 and merged up to this repository's main (torch 2.13 / cu130, ROCm 10 on Windows, the database layer). Every new upstream win32 marker is narrowed to x64 so Windows ARM64 keeps NVIDIA's torch in every extra.

Invoke could not be installed on Windows on ARM64 (NVIDIA's RTX Spark developer preview): every marker treated win32 as x64, the lockfile carried only win_amd64 wheels, and several dependencies have no ARM64 build at the pinned versions. With this PR, uv sync --frozen and uv pip install on a native ARM64 Python 3.12 produce a working install with NVIDIA's torch 2.14.0+cu134 (from https://pypi.nvidia.com/nvtorch_oot/), CI proves it on GitHub's windows-11-arm runners, and other platforms resolve as before apart from two changes: bcrypt 5 replaces passlib, and the xpu extra moves to the bitsandbytes version the other extras already use.

How it fits together:

• pyproject.toml forks the packages whose ARM64 wheels start at newer versions (numpy 2.3, onnxruntime 1.24, bitsandbytes 0.50.1, PyWavelets 1.10; main's onnx 1.21.0 and sentencepiece 0.2.2 already have ARM64 wheels, and its move to PyJWT removed the cryptography 46.0.x pin python-jose needed) behind sys_platform == 'win32' and platform_machine == 'ARM64', routes torch/torchvision to NVIDIA's index there for every extra, and sets uv's required-environments so a wheel-only package without an ARM64 wheel fails at lock time.
• OpenCV, blake3 and tiktoken have no ARM64 wheel anywhere. A new dispatch-only workflow builds them on windows-11-arm and publishes them, with a flat index.html, as assets of release win-arm64-wheels-v1 in this repository (byte-identical copies of the wheels first built in the now-archived InvokeAI-7); pyproject consumes that page as a flat index and pins the exact versions so the manual --find-links path prefers the wheels over PyPI's sdists. Pull-request runs of that workflow only build and verify. Published tags are immutable because the lockfile pins the assets by hash: a dispatch must name a new release_tag (no default), the publish job runs in the win-arm64-wheels environment, and it never replaces an asset — it uploads only names the release lacks, skips identical ones (so "Re-run failed jobs" can finish an interrupted upload) and fails on any other difference.
• scripts/check_aarch64_lock.py becomes scripts/check_platform_locks.py: linux/aarch64 still requires every extra to carry its own aarch64 torch pin, and win32/ARM64 additionally requires torch from NVIDIA's registry and walks the whole dependency closure for win_arm64 wheels (uv's required-environments cannot see packages that also publish an sdist). The closure check blocks: every dependency bump must keep a win_arm64 (or pure) wheel; .github/AGENTS.md describes the options when it fires.
• Runtime: mediapipe (no solutions API on ARM64) and umap/numba (no wheel for 3.12) are excluded there; the face nodes import mediapipe lazily and name the platform when it is the cause, the image map falls back to a PCA projection on Windows ARM64 only (elsewhere a failed umap import is a failed fit; the stored params record the method, so a PCA map is refit once umap imports), password hashing calls bcrypt directly (passlib 1.7 cannot use bcrypt >= 4.1, the first release with ARM64 wheels), the bitsandbytes guards survive a build whose native library fails to load and log the real error once (the loaders' error carries it), and startup logs the torch build and host architecture. One shim is specific to NVIDIA's build: without a CUDA device it dies with an access violation in torch.cuda.is_current_stream_capturing(), which transformers calls on every forward pass, so on such a machine the app (and the test session) make it answer False.
• Docs cover the native interpreter request (cpython-3.12-windows-aarch64-none), the install options, and the platform's feature differences.

The launcher needs a companion change (arch detection, aarch64 uv, native Python request, arm64 package build); it is prepared but not part of this PR. pins.json needs no change: the launcher's bootstrap path installs from the release's own lockfile.

Related Issues / Discussions

NVIDIA RTX Spark Developer Preview: https://forums.developer.nvidia.com/t/rtx-spark-developer-preview/377106

QA Instructions

After merging this repository's main (Linux, worktree venv):

• uv lock / uv lock --locked: resolved; against main's lock the changes are the ARM64 forks plus bcrypt 3.2.2 → 5.0.0 and passlib removed.
• check_platform_locks.py uv.lock → linux/aarch64 OK, win32/ARM64 OK (111 packages per ARM64 universe, 144 with test, none without a wheel); check_pins.py → consistent.
• pytest tests/backend/util/test_devices.py tests/app/services/auth tests/app/services/users tests/app/services/shared tests/app/util tests/test_check_pins.py tests/test_check_platform_locks.py tests/backend/util/test_rocm_sdpa_head_dim_guard.py → 1705 passed, 21 skipped. ruff check / ruff format --check clean.

After the first review round (Linux, worktree venv):

• uv lock --locked, check_pins.py and check_platform_locks.py pass; moving the wheel index changes only URLs in uv.lock (identical modulo the repository name).
• pytest -n 4 over the touched areas (image index/map, routers, devices, quantization, patches, model loading, imports, platform locks, wheel publish) → 2043 passed, 153 skipped; the image-map suites again with Windows ARM64 simulated (platform forced, umap/numba hidden) → 346 passed, 1 skipped.
• The new tests were checked against the unfixed code: each fails there (umap fallback scope, method staleness in the worker and the router, cache leak, float64 copy, shim install, bnb error text, per-extra aarch64 pin, duplicate output).

Before the move (InvokeAI-7, Linux, worktree venv):

• uv lock then uv lock --locked: resolved; the diff vs main is the ARM64 forks, bcrypt 3.2.2 → 5.0.0, passlib removed, and bitsandbytes 0.49.2 → 0.50.2 for the xpu extra on Linux x86_64 and Windows x64 (verified by parsing both locks: no other package's version or wheel list changed).
• uv run --no-project --with 'packaging<26' --python 3.12 scripts/check_platform_locks.py uv.lock → linux/aarch64 OK, win32/ARM64 OK (115 packages in each ARM64 universe, none without a wheel).
• python3 scripts/check_pins.py → consistent.
• pytest tests/app/services/auth tests/app/services/image_index tests/app/routers/test_utilities.py tests/backend/image_util tests/test_imports.py tests/test_check_platform_locks.py tests/test_check_pins.py → pass.
• Full suite (pytest -n logical): 8250 passed; 13 failures are the same GPU tests that fail identically on an untouched main checkout because both GPUs on this machine were at 99%/82% use (HIP out-of-memory in test_flux2_working_memory, test_model_cache_*, test_pid_chunked_equivalence), plus the repo-lock check that passes since the lock was regenerated.
• ruff check / ruff format --check clean; pnpm -C docs build → 326 pages, no new warnings.

GitHub (InvokeAI-7):

• build-win-arm64-wheels: OpenCV, blake3 and tiktoken built and smoke-imported on windows-11-arm; release win-arm64-wheels-v1 carries the three wheels and index.html; the publish job's uv pip compile --find-links <page> verification passed. (First OpenCV attempt failed: MSVC has no __fp16, which the DNN Winograd kernel uses once NEON_FP16 is in the CPU baseline; fixed by pinning the baseline to NEON + DOTPROD dispatch.)
• PR CI on the final commit: every check green. The py3.12: windows-arm64 lane installs from the lock on a native ARM64 interpreter (platform.machine() == "ARM64" asserted), imports the self-built wheels, and runs the full suite: 14,608 passed, 239 skipped, 8 xfailed in 31 minutes (run 39166449058; 37 minutes end to end, against a 60-minute job timeout, with a 600 s per-test timeout). Its runs surfaced these ARM64-only failures, all fixed: the stream-capture crash above (which had taken an xdist worker down and left ~2,500 tests unrun); torch 2.14 rejecting an SDPA test's explicit mask combined with is_causal=True; a frame fixture that overflowed uint8 (numpy 2 no longer wraps); a MediaPipe gate test that assumed a non-ARM64 host; a leaked ffmpeg (Chocolatey's ffmpeg is a launcher shim, and cancellation killed only the shim, leaving an endless test source pinning the runner's CPU); and frame counts doubled for clips whose soundtrack outlasts the video (the self-built OpenCV has no FFmpeg backend, and Media Foundation counts frames from the container duration).

Not verified (no hardware): CUDA generation on sm_121, cuDNN SDPA priority on this torch build, cudaMallocAsync acceptance in configure_torch_cuda_allocator(), bitsandbytes 0.50.2's CUDA kernels, VRAM accounting on unified memory. The docs mark the platform as a developer preview.

Review

Material findings resolved before the move: the wheel workflow's pull-request trigger could republish non-reproducible wheels over the assets the lock pins by hash (publishing is now dispatch-only under a concurrency group, and wheels are verified before upload); a top-level import mediapipe would have failed test_imports on ARM64; the woff2 font test needs brotli, absent on ARM64; the checker's closure walk skipped a package's extras once a plain edge had reached it (fixed, with a fixture that fails against the old walk); the MediaPipe error attributed every import failure to the platform; the admin guide still generated password hashes with passlib; bitsandbytes guards catch (ImportError, OSError, RuntimeError) rather than everything; the ARM64 test lane no longer caches uv's store. An old-vs-new lock comparison across the non-ARM64 platforms showed no change beyond bcrypt/passlib.

First review round (joshistoast, Pfannkuchensack): the wheel workflow could still replace pinned assets on a dispatch with default inputs, and the wheels lived in an archived repository (both fixed as above); the PCA fallback applied on every platform, was never refit once UMAP became available, and copied the whole matrix to float64; test guards for mediapipe, numba and brotli applied on every platform; one test leaked a cached None for UMAP into later tests, and the stream-capture shim was installed for the whole Linux test session; the bitsandbytes guards dropped the real error; the aarch64 check stopped requiring per-extra fallback pins and printed lines twice. All fixed with regression tests.

After the merge, the ARM64 cryptography pin had nothing left to serve (PyJWT does not need it) and was removed. Second review round (Pfannkuchensack): the lane timeout and the merge with main (fixed as above), plus the projection-method check (now one-way, so a UMAP map survives on a host without umap), the thread hop on every /points call, legacy-hash vectors (>72 bytes, $2a$/$2y$), and test and docs nits. Approval round (Pfannkuchensack): one shared process-tree kill that survives a denied suspend(), a deterministic shim test, a WOFF2-unsupported message, bcrypt bounded <6, and Chocolatey's ffmpeg pinned. Remaining risks: GPU behaviour on sm_121 is unverified (see QA), and the ffmpeg probes outside run_ffmpeg_process (and main's encode_rgb_frames, from #9856) still kill only a shim on timeout; they are left for a follow-up.

Compatibility / Rollout

• The published wheel's metadata now carries ARM64-only exact pins; other platforms' resolution changes only as listed below.
• All platforms: passlib is replaced by direct bcrypt 5 calls for password hashing (passlib 1.7 cannot use bcrypt >= 4.1); the requirement is bounded <6 until a new major is tested. Worth a changelog line.
• All platforms: the xpu extra now resolves bitsandbytes 0.50.2 on Linux x86_64 and Windows x64, the version the cpu, cuda and rocm extras already use (it was 0.49.2 only for xpu).
• Stored password hashes are unaffected: bcrypt verifies the $2b$12$ hashes passlib produced (covered by a regression test), and new hashes use the same cost factor.
• Bumping a self-built wheel means dispatching the workflow with a new release tag and updating the tag and pins in pyproject.toml together, then re-locking; never overwrite a tag that a tagged Invoke release's lock names.
• The launcher change ships separately; until then Windows ARM64 users install manually (documented). An existing launcher install on ARM64 runs an emulated x86_64 Python, which CPython 3.12 reports as ARM64, so it selects the ARM64 packages and cannot update to this release. The release notes should point those users to the new FAQ entry (reinstall with a native Python); ideally the launcher fix ships first.
• Risk: NVIDIA's nvtorch_oot index carries a single torch build (2.14.0+cu134); if it is withdrawn, the ARM64 lock breaks until a re-lock.
• Maintainers: add required reviewers to the win-arm64-wheels environment so a wheel dispatch cannot publish without approval.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Meaningful regression coverage added / updated where needed; obsolete tests/code removed
• Persisted-state and API changes include required migrations / compatibility validation
• Relevant performance/efficiency opportunities considered; material claims have evidence
• Material review findings resolved and relevant checks rerun
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[lstein]

@Pfannkuchensack thanks for the thorough review. I confirmed each point against the code before fixing it. Everything is in 3752285..9a2823d, and each fix has a regression test that fails against the old code.

Before merge

1. Archived repo: fixed. win-arm64-wheels-v1 now exists in this repo. It holds byte-identical copies of the three wheels (same SHA-256) and an index.html regenerated for this repo. pyproject.toml, manual.mdx and uv.lock point at it. With the repo name normalized, the lock diff is identical, so only URLs changed.

2. Republishing over the pinned tag: fixed as you suggested. Details are in the reply to joshistoast's comment:

   • release_tag is required and has no default;
   • assets are never replaced, but identical ones are skipped, so "Re-run failed jobs" can still finish an interrupted upload;
   • there's a top-level permissions: contents: read, and the publish job runs in a win-arm64-wheels environment.

   I went with "refuse anything that differs" instead of a separate "resume upload" input. A rerun of only the publish job reuses the same build artifacts, so their digests match and the rerun goes through; any fresh build won't match. Once this merges, required reviewers still have to be added to that environment in the repo settings.

Should fix

3. x64 Python on Windows ARM64: confirmed, and markers can't fix it. I added a FAQ entry, "Updating fails on Windows on ARM64…", which says to reinstall with a native Python, and linked it from the system requirements. The PR description's rollout section now says the release notes must point to it and that the launcher fix should ideally ship first.
4. bitsandbytes for xpu: confirmed. It's a side effect of the fork unification: the xpu extra now gets 0.50.2 on Linux x86_64 and Windows x64, the version cpu, cuda and rocm already resolve there. I kept 0.50.2 rather than pinning back to 0.49.2, since x64 CI already imports that version, and noted the change in the description. I have no XPU hardware to check its kernels on.
5. PCA fallback: all three sub-points fixed.
   • _umap_class() falls back only on Windows ARM64. Elsewhere the ImportError propagates, so the existing failed-fit and retry path handles it. functools.cache doesn't cache the raise, so a repaired install is picked up.
   • projection_params() records method. Both the worker's cache check and /points treat a stored PCA map as stale once umap imports, even when the scope is unchanged. Rows written before this change are read as umap, since main has no fallback. Maps of 3 points or fewer, and failed (empty) maps, are exempt, so the retry budget and refit loops behave as before.
   • PCA now streams a D×D float64 scatter matrix in 8192-row chunks and uses eigh, with a fixed sign. A tracemalloc test asserts that peak memory stays under half of an N×D float64 copy.
6. Weakened test guards: fixed. Allowing mediapipe_face_common to fail import, and the numba and brotli skips, now apply only when is_windows_arm64() is true. That's a new helper in invokeai/backend/util/host_platform.py, which the mediapipe gate uses too. With numba or brotli hidden on Linux, those tests now fail instead of skipping.
7. Test isolation: both problems fixed.
   • The umap-hiding fixture clears _umap_class's cache on exit as well as on entry. A test checks that nothing cached survives the block.
   • tests/conftest.py installs the stream-capture shim only on Windows ARM64. The shim test now installs over a fresh, unshimmed stand-in and asserts the function was actually replaced.
   • Our own review found one more leak: test_apply_monkeypatches_installs_the_guard left the shim installed on a CUDA build without a GPU. That test now restores the original.
8. bitsandbytes errors: fixed. A new invokeai/backend/quantization/bnb_import.py records the first failure from any guard. It logs once, unless bitsandbytes simply isn't installed, as on macOS. The loaders' error now reads "bitsandbytes is installed but failed to load: …" with the real cause.

Minor

• Contributor runbook: added as "Windows ARM64 lock gate" in .github/AGENTS.md. It covers pinning the last version with a wheel, dropping the package by marker with a degraded path, or self-building it under a new tag. On block versus warn: I'm keeping the closure check blocking. A warning would let a release ship a lock that fails partway through installing on every ARM64 machine, and the contributor doing the bump is the person best placed to choose one of those options.
• NVIDIA index: noted as a risk in the description.
• Changelog: the PR title now names the bcrypt 5 change, and the compatibility section flags that it affects every platform.
• aarch64 check: you were right on both counts.
  • The base torch==2.7.1 edge happened to satisfy every extra, so removing the cpu extra's aarch64 edges passed the check. linux/aarch64 now requires each extra to carry its own pin, and your mutation fails again.
  • The duplicate lines came from that same base-plus-extra edge pair, and win_arm64 printed them twice as well. Each package now prints once per extra.
• Stale text: fixed the onnx extras comment, the checker docstring and --help, and the fixtures. The fixtures now model the blake3 index fork and transformers→tokenizers in place of cryptography and python-jose.
• patchmatch: patchmatch.mdx now says the patchmatch failed to load or compile error at startup is expected on ARM64.
• Commit trailers: I'll strip them when squashing.

Checks:

• uv lock --locked, check_pins.py and check_platform_locks.py are green.
• The touched suites pass on Linux.
• The image-map suites also pass with Windows ARM64 simulated, with the platform forced on and umap and numba hidden. That's the lane where our own review caught four tests that had assumed x64.

[Pfannkuchensack]

Thanks for the quick turnaround. I re-reviewed head 9a2823d. Almost everything from my first pass is resolved:

• Wheels: they're hosted here now. win-arm64-wheels-v1 digests match the lock, and only URLs changed.
• Publish workflow: it requires a new tag and --clobber is gone. plan_release_upload.py refuses on any difference, and the protected win-arm64-wheels environment and permissions: contents: read are in place.
• Image map: the PCA fallback is ARM64-only, records the method in the params, refits once UMAP imports, and uses the D×D scatter matrix. That's about 0.17 GB extra at 300k points, down from 2.5 GB at 100k before.
• bitsandbytes: guards now log the real failure and carry it into the loader error.
• Tests and checker: x64 skips are confined to ARM64, the conftest shim only installs there, the PCA cache leak is gone, and the aarch64 check again requires a per-extra pin.
• Disclosure and docs: the bitsandbytes xpu change is disclosed, and the FAQ and patchmatch docs are added.

Locally, ruff, uv lock --locked, check_platform_locks.py and check_pins.py pass, along with the focused tests (also with bcrypt 5). The old-vs-new lock comparison again differs only in bcrypt/passlib and bitsandbytes for the xpu extra.

Still blocking

1. The py3.12: windows-arm64 lane times out on both heads. Runs 38663611897 and 38676871057 were both cancelled at the 60-minute job timeout at about 74 %. Both show the same 8 failures on the 73 % progress line (...FFF.FFF.......FF...), and the progress has 5–8 minute gaps, so something is hanging or very slow. Because the job is cancelled before pytest prints its summary, the failing tests can't be identified from the log. Adding -rf, a junit report, or a per-test --timeout would surface them. The lane numbers in the description ("every check green", 13 min) come from InvokeAI-7, not from this repo.
2. The merge with current main breaks the test suite. The textual conflict is only in .github/AGENTS.md (after #9863). The semantic one is in tests/conftest.py: since #9859, main's _cheap_password_hashing and production_password_hashing patch password_utils.pwd_context, which this PR removes. After the merge, collection fails. Your hash_password reads BCRYPT_ROUNDS on each call, so the fix is simple: patch password_utils.BCRYPT_ROUNDS to 4 in the autouse fixture and restore 12 in production_password_hashing. Also keep test_hashing_cost_stays_at_a_defensible_work_factor on production_password_hashing. When resolving python-tests.yml, don't copy windows-cpu's not platform_independent marker to the ARM64 entry; ARM64 has its own dependency set, so it should run those tests.

Minor

• .github/AGENTS.md placement: since #9863 it's a terse policy file. One or two policy lines (the gate blocks; the options are a fork pin, a marker exclusion with a system-requirements entry, or a self-built wheel) would fit. The dispatch/retag/relock procedure is better kept in the workflow header or in docs/.../development/.
• Two-way method check: projection_method_is_current treats any mismatch as stale. On ARM64, a valid UMAP map (a database copied from x64, or a legacy row without the field) gets replaced by PCA. Treating only stored pca with UMAP available as stale avoids that.
• Thread hop: image_map.py now calls asyncio.to_thread(projection_method_is_current, …) on every /points call on every platform. Doing it inline unless is_windows_arm64() would skip that.
• Partial uploads: the workflow header says "re-run failed jobs" completes an interrupted upload. An upload that dies mid-asset can leave a digest-less asset, which the planner (correctly) refuses, so a manual delete is needed. A note in the header would help.
• Still open from last time:
  • test vectors for >72-byte and $2a$/$2y$ legacy passwords;
  • the "protobuf mismatch" comment in mediapipe_face/__init__.py (that failure raises TypeError, not ImportError);
  • the mediapipe gate test patching global sys.platform (patching mediapipe_face.is_windows_arm64 would be narrower);
  • the production shim scope and docstring;
  • the tab labels in manual.mdx;
  • ${{ matrix.* }} interpolated into a run: step in the wheel workflow.

[lstein]

Thanks for the second pass. Here is where each item stands; head is 44b63e6876.

Blocking

1. The windows-arm64 lane timeout. I added a per-test --timeout and -v for diagnosis, then sampled the runner's CPU per process. Memory stayed at about 10 GB free throughout.
   • About 20 minutes in, an orphaned ffmpeg -f lavfi -i anullsrc -f null - held the 4-vCPU runner at 100% for the rest of the job. It came from test_cancellation_kills_a_running_ffmpeg.
   • The cause: on ARM64, imageio_ffmpeg.get_ffmpeg_exe() resolves to the ffmpeg on PATH. Chocolatey's ffmpeg is a launcher shim that runs the real binary as a child, and run_ffmpeg_process killed only the shim.
   • The 8 failures (the sqlite cancel/retry tests) and the 300 s timeouts (the import-boundary subprocess tests, which take 14 s on an idle runner) all followed from the starved CPU. The 5–8 minute gaps in progress did too.
   • Users hit the same bug: any ARM64 install with a Chocolatey or Scoop ffmpeg leaked one on every cancel or timeout. 71b93a424a now suspends the shim, kills the process tree, and routes the upload conversion through the same runner. A regression test runs a Python launcher shim, and that test fails against the old kill.
   • With the leak gone, the suite finished in 23 minutes with 4 real failures, all in test_video_frame_extract_range_audio_splice.py. The self-built OpenCV wheel has no FFmpeg backend, and Media Foundation derives CAP_PROP_FRAME_COUNT from the container duration. A soundtrack longer than the video therefore doubled the frame count. 54dc0e6464 makes the decode worker count video packets with ffmpeg (-c copy -f framecrc) whenever getBackendName() != "FFMPEG". Other platforms are unchanged.
   • The lane now keeps --timeout=600, so a hang becomes a traceback and the summary still prints. Its job timeout is 60 minutes, about 1.6× the slowest measured job (37 min). Run 39166449058 on 88f951826e (the last commit only raises that timeout): 14,608 passed, 239 skipped, 8 xfailed in 31 min; every other check is green.
2. The merge with main. 8fa82de6dd merges current main:
   • The conftest.py fixtures patch password_utils.BCRYPT_ROUNDS (4 for the autouse fixture, the production value in production_password_hashing).
   • test_hashing_cost_stays_at_a_defensible_work_factor stays on production_password_hashing.
   • The ARM64 matrix entry has no pytest-markers, so it runs the platform_independent tests.
   • I took main's uv-lock-checks.yml setup-uv pin and kept the ARM64 skip in test_umap_import.py.

Minor

• .github/AGENTS.md: it's main's terse file plus one policy line: the gate blocks; the options are a fork pin, a marker exclusion with a system-requirements.mdx entry, or a self-built wheel. The procedure lives in the build-win-arm64-wheels.yml header.
• Two-way method check: projection_method_is_current is now one-way. Only a stored pca map where UMAP imports counts as stale, so a UMAP map, or a legacy row without the field, survives on a host without umap. There is a test for both cases.
• Thread hop: image_map.py calls the check inline, and uses asyncio.to_thread only when is_windows_arm64().
• Partial uploads: the workflow header now says an upload that dies mid-asset can leave a digest-less asset, which has to be deleted by hand before re-running.
• Still open from last time:
  • Legacy-hash vectors made with passlib 1.7.4 + bcrypt 3.2.2: over 72 bytes (ASCII, and UTF-8 split at the boundary), $2a$ and $2y$. Each one checks that the password verifies and that a one-character change does not.
  • The mediapipe_face/__init__.py comment now says "a missing native library".
  • The gate test patches mediapipe_face.is_windows_arm64.
  • The stream-capture shim returns early unless is_windows_arm64(). Its docstring says NVIDIA's ARM64 build crashes where others raise. A test covers the non-ARM64 no-op.
  • The manual.mdx tab is "Windows on ARM64", matching step 8.
  • The wheel workflow passes matrix.* values through env:. Every setup-uv use is pinned to the hash.

Known gap

The two ffmpeg probe calls (probe_media_streams, _probe_audio_sample_rate) and the LTX-2 conditioning helper still use subprocess.run(timeout=...), so behind a shim a timeout leaves the real ffmpeg running. These are short probes that only time out on hostile input. I'd rather fix them in a follow-up than widen this PR. I'll also update the description's lane numbers to this repo's run.

[Pfannkuchensack]

Thanks, that all checks out on 44b63e6876:

• The windows-arm64 lane is green (36 min against a 60 min job timeout).
• The conflict resolution looks right.
• The packet-count fix for the 4 video failures is sound, and its test is a good regression test.
• Leaving the probe timeouts for a follow-up is fine with me.

A few small things left, none of them big:

1. The shim regression test can pass without exercising the kill. In test_cancellation_kills_the_ffmpeg_a_launcher_shim_started, Popen is patched before _ffmpeg_exe() runs. When the test is the first ffmpeg lookup in its worker, imageio_ffmpeg's lru_cached get_ffmpeg_exe() validates the binary with a -version subprocess, and that call goes through the fake shim too. The shim writes the probe's PID, is_canceled() fires on the first poll, and the test then finds that PID already gone and passes. With the cache cold, it also passes against the old proc.kill(); proc.wait(). It only fails once the cache is warm, so whether CI actually checks the fix depends on xdist ordering. Calling video_audio._ffmpeg_exe() before the patch fixes it. Alternatively, the shim could record the PID only for the anullsrc command.
2. The new process-tree kill duplicates an existing helper. _kill_process_tree (video_audio.py:49) duplicates video_thumbnails._terminate_process_tree. Yours is the better one, since it suspends the parent before listing the children, so I'd fold the two together. One detail: suspend() and children() share a try, so if suspend() raises (for example AccessDenied), the child list is dropped and the real ffmpeg survives, even though children() alone would have worked.
3. The new packet count has no timeout of its own. _ffmpeg_packet_count calls subprocess.run with no timeout. That's fine in practice, because the worker's 30 s budget and _terminate_process_tree bound it and kill the descendants. Just note that on ARM64 it now demuxes the whole file inside that budget, so a very large upload on slow storage could hit it.
4. History cleanup before merge: squash away the two temporary diagnostics commits, and drop the Co-Authored-By: Claude / Claude-Session: trailers from the squash message.

Separately, for the launcher: OSSign/invoke-ai-launcher#3 was merged at 0a9ff2d without the VS 2022 Build Tools step or npm_config_msvs_version that launcher #159 needed on windows-11-arm. Its build-arm64 job also runs npm ci before the "does this ref support ARM64" check. So I'd expect the next launcher tag, even one without #159/#160, to fail in the arm64 job, which means release never runs and no Windows release is published, x64 included. It's worth porting the toolchain fix (and moving the support check right after checkout) before the next launcher tag.

[Pfannkuchensack]

Re-reviewed 44b63e6876, still on x64 only. Approving: both blockers from my second pass are fixed, and everything else from my first two passes is in. Two small items from my last comment are still open (inline), plus a few nits.

Earlier points

Point	Status	Evidence
joshistoast's asset replacement / my P1-2	Resolved	release_tag is required with no default, --clobber is gone, plan_release_upload.py refuses differing or digest-less assets, permissions: contents: read, publish job in win-arm64-wheels
P1-1 archived wheel repo	Resolved	The index points at win-arm64-wheels-v1 in this repo
P1-3 x64 Python on ARM64	Resolved (docs)	FAQ entry and system requirements; release order in the description
P1-4 bitsandbytes for xpu	Disclosed	The only non-ARM64 change besides bcrypt/passlib (see below)
P1-5 PCA fallback	Resolved	ARM64 only, method recorded, chunked D×D scatter + eigh
P1-6 test guards	Resolved	mediapipe, numba and brotli allowances gated on is_windows_arm64()
P1-7 test isolation	Resolved	_umap_hidden clears on exit, conftest shim ARM64-only, shim test uses a fresh stand-in
P1-8 bitsandbytes error	Resolved	bnb_import.py; the loaders raise the recorded cause
P1 minors	Resolved	Runbook line, NVIDIA risk, title, per-extra aarch64 pin, stale text, patchmatch
P2-1 ARM64 lane timeout	Resolved	The orphaned anullsrc behind Chocolatey's shim is fixed by the tree kill, the doubled frame counts by the packet count; lane green at 36 of 60 min
P2-2 merge with main	Resolved	BCRYPT_ROUNDS fixtures, cost test on production_password_hashing, ARM64 entry runs platform_independent
P2 minors	Resolved	One-way method check, inline check off ARM64, partial-upload note, legacy vectors, comment, gate test, shim scope, tab label, matrix.* through env:
P3-1 shim test	Open	Inline
P3-2 tree kill	Open	Inline
P3-3 packet-count timeout	Fine as is	Bounded by the worker's 30 s budget and its tree kill
P3-4 history	At merge	See below

Before merging

• Squash away the two "temporarily sample" commits and drop the Co-Authored-By / Claude-Session: trailers from the squash message.
• joshistoast's CHANGES_REQUESTED review is still on record; it needs their re-review or a dismissal.
• Add required reviewers to the win-arm64-wheels environment in the repository settings. Until then a dispatch publishes without approval.

Verified

• Passwords: I generated hashes with the former passlib 1.7.4 + bcrypt 3.2.2 code for nine passwords (ASCII, unicode, emoji, exactly 72 and 73 bytes, over 72 with a UTF-8 character split at byte 72, 200 bytes), each as $2b$, $2a$ and $2y$. All 27 verify on this branch under bcrypt 5.0.0, a changed first character never does, and hashes from this branch verify under the old passlib code, so a rollback keeps working. Inputs over 72 bytes don't raise (bcrypt 5's own hashpw/checkpw do, so the truncation is load-bearing), and empty or malformed hashes return False. One small difference: passlib rejected passwords containing NUL; bcrypt 5 hashes them in full.
• Lockfile: comparing base and head per platform (Linux x86_64/aarch64, Windows x64, macOS arm64/x86_64, every extra combination; versions, sources and hashes), the only changes are bcrypt 3.2.2 → 5.0.0, passlib removed, and bitsandbytes 0.49.2 → 0.50.2 for xpu on Linux x86_64 and Windows x64.
• ruff, check_platform_locks.py and the focused suites pass locally; the auth and multiuser suites also under bcrypt 5.

Follow-ups, not blocking

• #9856 on main: video_smart_concat.encode_rgb_frames landed after this branch's last merge and kills only its direct child on cancel or timeout. Behind a shim on ARM64 the real ffmpeg keeps encoding until it reads EOF on stdin, so it ends on its own, and the tests mock it. Worth adding to the follow-up with the two probes.
• nit: the description still says in a few places that nothing changed outside ARM64 ("exactly as before apart from one deliberate change", "no other package's version or wheel list changed"), while the compatibility section correctly lists the xpu bitsandbytes bump.

[lstein]

Thanks for the approval. Head is f97360db19; CI is running.

• Shim test (P3-1): the test calls video_audio._ffmpeg_exe() before patching Popen, so the cold-cache -version probe never goes through the shim. It now fails against the plain proc.kill(); proc.wait() when run on its own.
• PID reuse in the test: it keeps the psutil.Process captured while ffmpeg is alive. is_running() and kill() check its create time, so they can't reach a process that reused the PID.
• Tree kill (P3-2): invokeai/app/util/process_tree.py has one kill_process_tree, used by run_ffmpeg_process and by video_thumbnails._terminate_process_tree. Only the latter tries killpg first, since only its worker has its own session.
  • suspend() is best-effort, and the children are listed in their own try. A new parametrization denies suspend(); it fails against the previous helper.
  • The helper returns early for an already-reaped proc, whose PID may be reused.
  • When psutil.Process(proc.pid) raises because the shim has exited, children are not looked for. Without the shim's create time, they can't be told apart from unrelated processes that hold a reused parent PID. Shims wait for their child, so an exited shim means an exited ffmpeg. The docstring says so.
• Chocolatey: pinned to ffmpeg --version 9.0.2, the version every green run used.
• bcrypt: bcrypt>=4.1,<6; the lock is otherwise unchanged.
• WOFF2 without brotli: an ImportError while reading a .woff2 now raises FontValidationError("WOFF2 fonts are not supported on this platform"). There is a test for it (forcing woff2.haveBrotli off), and the system-requirements line is updated.
• Description: the "exactly as before" and "no other package changed" wording now names the xpu bitsandbytes bump.
• Lane as critical path: agreed. It stays blocking while the platform is new. Narrowing it to the merge queue, main, releases and dependency changes is a follow-up.
• Follow-up: encode_rgb_frames from Add GOP-aware smart rendering to video concatenation #9856 goes on the list with the two probes and the LTX-2 helper. One more for that PR: only the root is suspended, so a grandchild started between the listing and the kill can still escape. On Windows a Job object would close that gap.
• Before merge: I'll squash with a clean message, which drops the diagnostics commits and the trailers. The joshistoast re-review or dismissal and the win-arm64-wheels environment reviewers are on the maintainers' side.