Repository navigation
fix: CON-1449/1458/1455 recoverable ComfyUI startup instead of silent restart loops - #172
Open
mariiachekmasova-runpod wants to merge 23 commits into
Open
mariiachekmasova-runpod wants to merge 23 commits into
mariiachekmasova-runpod wants to merge 23 commits into
Conversation
mariiachekmasova-runpod
marked this pull request as ready for review
October 1, 2026 09:48
nick4fake
requested changes
Oct 2, 2026
| # and let the `wait` below return so it can exit before the cleanup. | ||
| if [ -n "${COMFY_PID:-}" ] && kill -0 "$COMFY_PID" 2>/dev/null; then | ||
| kill "$COMFY_PID" 2>/dev/null || true | ||
| return |
Contributor
There was a problem hiding this comment.
return 0
the shutdown handler ends with a bare return. When the pod is stopped while ComfyUI is running, the script exits with code 143 and skips the shutdown steps.
- The shutdown isn't logged.
- Jupyter and FileBrowser are killed instead of being stopped cleanly
Contributor
Author
There was a problem hiding this comment.
Fixed. The handler now does everything itself instead of leaving work for after
wait: it logs the shutdown, sends SIGTERM to ComfyUI and waits for it to
finish, then stops Jupyter and FileBrowser and exits 0. SHUTTING_DOWN,
stop_services and the block after wait are gone.
A restart now logs the "Pod is shutting down" line again, with no crash banner,
and ComfyUI exits first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
A ComfyUI pod could end up permanently broken, or stuck restarting, while the log
said nothing for minutes. Five separate defects, all on the start path.
Interrupted first-boot copy left a workspace that claimed to be complete
(CON-1449). The copy used
cp -rand wrote the bundle manifest along witheverything else, so a copy stopped part-way — minutes long on a network volume —
could leave a partial tree that the next boot accepted as current. The result
was
ModuleNotFoundErroronapp,comfy.optionsor a missingmain.py, andit was permanent: the manifest matched, so nothing ever repaired it, and the
broken tree followed the volume across pods. The tree is now staged under
ComfyUI.incompleteand renamed into place, so the directory exists only oncethe copy finished; an interrupted copy is resumed with
rsyncon the next boot.A half-created venv was treated as usable (CON-1458). The guard tested only
that
.venv-cu128existed. Whenpython -m venvfailed before writingbin/activate,sourceaborted the script underset -eand the containerrestarted straight into the same failure, tearing down SSH, JupyterLab and
FileBrowser each cycle — the user sees a pod with no reachable port at all. The
check is now on
bin/activate, activation is hoisted out of the branches soevery path reaches it, and an unusable venv is recreated. One that holds
packages is moved to
.venv-cu128.broken.<date>rather than deleted.Nothing restarts the container silently any more. An ERR trap installed
once SSH/Jupyter/FileBrowser are up holds the pod with an explanatory banner
instead of exiting, and SIGTERM handling moved above the ComfyUI launch so a pod
stopped during setup still logs it. Both hold points background the
sleepandwaitfor it: bash runs no trap while waiting on a foreground command, so aplain
sleep infinityswallowed the stop signal until SIGKILL.Minutes of silence during startup (CON-1455). The workspace copy and the
pinned-package install now print a heartbeat every 10s and their elapsed time,
and a background poller announces
ComfyUI is READY on port 8188as soon as theport answers — 11s on a test pod, against 3m54s for ComfyUI-Manager's
"All startup tasks have been completed", which is what the README used to point
at and what users were waiting for before stopping the pod.
An OOM kill is named (CON-1452). Exit 137 with a non-zero cgroup
oom_killcounter now reports that the container ran out of RAM and prints thelimit, instead of the generic banner that sends people looking for a traceback
that cannot exist. A SIGKILL with the counter at zero is reported as such. The
rest of CON-1452 — memory sized off host RAM — is fixed upstream in ComfyUI
0.35 and arrives with #164.
Verification
Exercised on live pods, not only in CI: first-boot copy with the heartbeat;
resume after an interrupted copy (85s full copy vs 20s resume); leftover stage
directory removed; broken empty venv recreated; broken venv with packages moved
aside with the packages intact; pinned-package reinstall; an unexpected startup
failure holding the pod with SSH and Jupyter reachable and no restart; pod
stopped during setup, while ComfyUI runs, and after it dies — each logging the
shutdown; and a SIGKILLed ComfyUI producing the banner with the pod still up.
The two paths not reproduced on a pod are a real out-of-memory kill and a failing
python -m venv; both were exercised against the script's logic in isolation.