fix: back off when the watchdog rebuild is triggered by degradation - #399
Merged
Conversation
The rebuild loop reset backoff to 1s on any clean return from _event_loop, and a rebuild requested by FAIL_THRESHOLD consecutive COMErrors returns exactly that way. So a UIA stack that degrades immediately on every rebuild was retried once a second forever: a fresh COM client built and torn down per second, and one "WatchDog event pipeline degraded" warning per second. The exponential backoff only ever applied to the exception path. Reset backoff only after a run that survived HEALTHY_RUN_SECONDS, so an intermittent glitch after hours of healthy operation still retries promptly, while a persistently broken environment escalates to the 30s cap. Verified against the old code: twelve consecutive immediate degradations produced twelve 1.0s waits, where they now escalate 1, 2, 4, 8 ... 30. This is the log flood in #332 and the client churn behind it. It does not address the terminal E_UNEXPECTED crash, which is a native access violation no Python except can catch and needs the watchdog out of process. Refs #332
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| monkeypatch.setattr(watchdog_service.comtypes, "CoUninitialize", lambda: None) | ||
|
|
||
| watchdog = WatchDog() | ||
| monkeypatch.setattr(watchdog, "_create_uia", lambda: object()) |
This was referenced Aug 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #332. This is a bug in the recovery loop added by #334, not the terminal crash — see the caveat at the bottom.
The bug
_runresetbackoff = 1.0on any clean return from_event_loop, with the comment "clean exit (stopped or rebuild requested)". But a rebuild requested byFAIL_THRESHOLDconsecutiveCOMErrors returns exactly that way —_event_loopexits normally with_needs_rebuildset. Treating that as clean means the exponential backoff only ever applied to the exception path.The consequence in the environment #332 describes, where the focus pipeline is persistently degraded:
request_rebuild()fires_event_loopreturns, loggingWatchDog event pipeline degraded, rebuilding UIA clientat WARNINGIUIAutomationclient is built and immediately degrades againRepeat forever, once a second. That is the "thousands of identical lines" in the original report, plus a COM client constructed and torn down every second — churn that plausibly contributes to the state collapse rather than damping it.
The fix
Reset the backoff only after a run that survived
HEALTHY_RUN_SECONDS(60s). An intermittent glitch after hours of healthy operation still retries in one second, as intended; an environment that degrades the moment it rebuilds escalates 1, 2, 4, 8 … up to the existing 30s cap.Verification
Six tests in
tests/test_watchdog_backoff.pydrive_rundirectly with every COM touchpoint stubbed, capturing what it would have slept.Checked against the pre-fix code, which is how the diagnosis was confirmed rather than assumed:
Twelve consecutive immediate degradations produced twelve 1.0s waits before, and reach the 30s cap now. The exception path, the "long healthy run resets backoff" path, the stop path, and #334's contract that each cycle builds a fresh client with clean failure counters are all covered too.
Full suite: 587 passed.
What this does not fix
The terminal
-2147418113/E_UNEXPECTEDcrash from the original report is untouched. That is a native access violation duringPumpEvents, which no Pythonexceptcan catch, and isolating the server from it needs the watchdog moved out of process. This change reduces how hard and how often the watchdog hammers a collapsing UIA stack on the way there, and makes the degradation legible in the log instead of drowning it — but #332 should stay open for the out-of-process work.