Skip to content

fix: CON-1449/1458/1455 recoverable ComfyUI startup instead of silent restart loops - #172

Open
mariiachekmasova-runpod wants to merge 23 commits into
mainfrom
fix/CON-1449-ComfyUI
Open

mariiachekmasova-runpod wants to merge 23 commits into
mainfrom
fix/CON-1449-ComfyUI

Conversation

@mariiachekmasova-runpod

@mariiachekmasova-runpod mariiachekmasova-runpod commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

What this fixes

A ComfyUI pod could end up permanently broken, or stuck restarting, while the log
said nothing for minutes. Five separate defects, all on the start path.

Interrupted first-boot copy left a workspace that claimed to be complete
(CON-1449).
The copy used cp -r and wrote the bundle manifest along with
everything else, so a copy stopped part-way — minutes long on a network volume —
could leave a partial tree that the next boot accepted as current. The result
was ModuleNotFoundError on app, comfy.options or a missing main.py, and
it was permanent: the manifest matched, so nothing ever repaired it, and the
broken tree followed the volume across pods. The tree is now staged under
ComfyUI.incomplete and renamed into place, so the directory exists only once
the copy finished; an interrupted copy is resumed with rsync on the next boot.

A half-created venv was treated as usable (CON-1458). The guard tested only
that .venv-cu128 existed. When python -m venv failed before writing
bin/activate, source aborted the script under set -e and the container
restarted straight into the same failure, tearing down SSH, JupyterLab and
FileBrowser each cycle — the user sees a pod with no reachable port at all. The
check is now on bin/activate, activation is hoisted out of the branches so
every path reaches it, and an unusable venv is recreated. One that holds
packages is moved to .venv-cu128.broken.<date> rather than deleted.

Nothing restarts the container silently any more. An ERR trap installed
once SSH/Jupyter/FileBrowser are up holds the pod with an explanatory banner
instead of exiting, and SIGTERM handling moved above the ComfyUI launch so a pod
stopped during setup still logs it. Both hold points background the sleep and
wait for it: bash runs no trap while waiting on a foreground command, so a
plain sleep infinity swallowed the stop signal until SIGKILL.

Minutes of silence during startup (CON-1455). The workspace copy and the
pinned-package install now print a heartbeat every 10s and their elapsed time,
and a background poller announces ComfyUI is READY on port 8188 as soon as the
port answers — 11s on a test pod, against 3m54s for ComfyUI-Manager's
"All startup tasks have been completed", which is what the README used to point
at and what users were waiting for before stopping the pod.

An OOM kill is named (CON-1452). Exit 137 with a non-zero cgroup
oom_kill counter now reports that the container ran out of RAM and prints the
limit, instead of the generic banner that sends people looking for a traceback
that cannot exist. A SIGKILL with the counter at zero is reported as such. The
rest of CON-1452 — memory sized off host RAM — is fixed upstream in ComfyUI
0.35 and arrives with #164.

Verification

Exercised on live pods, not only in CI: first-boot copy with the heartbeat;
resume after an interrupted copy (85s full copy vs 20s resume); leftover stage
directory removed; broken empty venv recreated; broken venv with packages moved
aside with the packages intact; pinned-package reinstall; an unexpected startup
failure holding the pod with SSH and Jupyter reachable and no restart; pod
stopped during setup, while ComfyUI runs, and after it dies — each logging the
shutdown; and a SIGKILLed ComfyUI producing the banner with the pod still up.

The two paths not reproduced on a pod are a real out-of-memory kill and a failing
python -m venv; both were exercised against the script's logic in isolation.

@mariiachekmasova-runpod mariiachekmasova-runpod changed the title fix: CON-1449 fix: CON-1449/1458/1455 recoverable ComfyUI startup instead of silent restart loops Sep 29, 2026
Base automatically changed from feat/CON-1576-split-releases to main October 1, 2026 08:59
@mariiachekmasova-runpod
mariiachekmasova-runpod marked this pull request as ready for review October 1, 2026 09:48
# and let the `wait` below return so it can exit before the cleanup.
if [ -n "${COMFY_PID:-}" ] && kill -0 "$COMFY_PID" 2>/dev/null; then
kill "$COMFY_PID" 2>/dev/null || true
return

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return 0

the shutdown handler ends with a bare return. When the pod is stopped while ComfyUI is running, the script exits with code 143 and skips the shutdown steps.

  • The shutdown isn't logged.
  • Jupyter and FileBrowser are killed instead of being stopped cleanly

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. The handler now does everything itself instead of leaving work for after
wait: it logs the shutdown, sends SIGTERM to ComfyUI and waits for it to
finish, then stops Jupyter and FileBrowser and exits 0. SHUTTING_DOWN,
stop_services and the block after wait are gone.

A restart now logs the "Pod is shutting down" line again, with no crash banner,
and ComfyUI exits first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants