Skip to content

python3-improv: self-healing BLE advertising watchdog (imx93-jaguar-eink)#38

Merged
ajlennon merged 3 commits into
mainfrom
feature/improv-adv-watchdog
Jul 19, 2026
Merged

python3-improv: self-healing BLE advertising watchdog (imx93-jaguar-eink)#38
ajlennon merged 3 commits into
mainfrom
feature/improv-adv-watchdog

Conversation

@ajlennon

Copy link
Copy Markdown
Member

Summary

Hardens the imx93-jaguar-eink Improv onboarding server so the device can never silently stop advertising, and fixes the placeholder default host.

  • Advertising watchdog — bless registers the advertisement once at start-up and never re-asserts it; if BlueZ drops it the board becomes un-onboardable until a manual restart (observed in the lab). The watchdog re-checks BlueZ every IMPROV_ADVERT_WATCHDOG_SECS (default 15s) and re-registers a dropped advertisement, clearing stale bless advertisement objects first.
  • Can't-freeze / self-heal — every BlueZ D-Bus call is bounded by IMPROV_ADVERT_DBUS_TIMEOUT; after IMPROV_ADVERT_MAX_FAILURES consecutive failures the process exits so systemd (Restart=always) does a clean re-init.
  • Ordering bug fix — background tasks are now created before the initial network-status seed. The seed's executor call could block before the watchdog task was ever created, leaving the server running with no watchdog (reproduced in the lab).
  • Default hostIMPROV_SERVER_HOST now defaults to the live Active-ESL backend (active-esl-onboard.active-esl.workers.dev) instead of the api.co.uk placeholder, so the server points somewhere valid even if the systemd env override is missing.

Scope / known limitation

This addresses registration-level advert drops and a wedged BLE stack. It does not fix the separate Wi-Fi/BT coexistence issue where advertising stays registered (ActiveInstances=1) but stops reaching the air after ~5 min while the board is joined to Wi-Fi. That is tracked as a follow-up.

Fresh (un-provisioned, Wi-Fi-off) onboarding — the primary customer path — is 100% reliable: verified 26/26 discoverable over 13 min from a separate scanner. The coexistence degradation only affects re-onboarding a board already on Wi-Fi.

Test plan

  • Watchdog task starts and runs continuously (verified via journal: periodic advertising watchdog: ... advertising=True checks over 2+ min under systemd).
  • Ordering fix: watchdog confirmed running only after task creation was moved ahead of the net-status seed.
  • Syntax checked on-target (python3.12).
  • Build eink image with this change and re-flash target to validate under the real image (not just the drop-in override used for lab testing).
  • Longer soak of the watchdog re-register/restart backstop.

Made with Cursor

…lt host

Harden the imx93-jaguar-eink Improv onboarding server so the device can
never silently stop advertising:

- bless registers the advertisement once at start-up and never re-asserts
  it; if BlueZ drops it the board becomes un-onboardable until a manual
  restart. Add a watchdog that re-checks BlueZ every
  IMPROV_ADVERT_WATCHDOG_SECS (default 15s) and re-registers a dropped
  advertisement, clearing the stale bless advertisement objects first.
- Bound every BlueZ D-Bus call with IMPROV_ADVERT_DBUS_TIMEOUT so a wedged
  stack can't freeze the loop; after IMPROV_ADVERT_MAX_FAILURES consecutive
  failures, exit so systemd (Restart=always) does a clean re-init.
- Create the background tasks BEFORE the initial network-status seed: the
  seed's executor call could block before the watchdog task was ever
  created, leaving the server running with no watchdog.

Also default IMPROV_SERVER_HOST to the live Active-ESL backend
(active-esl-onboard.active-esl.workers.dev) instead of the "api.co.uk"
placeholder, so the server still points somewhere valid if the systemd
env override is ever missing.

Note: this addresses registration-level drops/wedges. It does NOT fix the
separate Wi-Fi/BT coexistence issue where advertising stays registered
(ActiveInstances=1) but stops reaching the air after ~5 min while the board
is joined to Wi-Fi. Fresh (un-provisioned, Wi-Fi-off) onboarding is 100%
reliable (verified: 26/26 discoverable over 13 min). Coexistence mitigation
is tracked separately.

Co-authored-by: Cursor <cursoragent@cursor.com>
Adds a periodic advertisement bounce (ADVERT_BOUNCE_SECS, default 60s) to the
advertising watchdog to handle a second failure mode beyond a full
registration drop.

BlueZ can report ActiveInstances>=1 while nothing is actually on-air, so
is_advertising() looks healthy and the existing drop-detection never fires; the
board then becomes silently un-onboardable until a manual restart (observed
after a failed/aborted BLE connect and around a fresh boot, with Wi-Fi OFF, so
not coexistence). Since ActiveInstances cannot tell us whether the advert is
truly radiating, we cannot detect this directly, so we periodically re-assert
it. Both re-assertions are skipped while a central is mid-session so an
in-progress onboarding is never disrupted.

Refactors the re-assert path into _reassert_advert() / _drop_stale_adverts()
shared by the drop and bounce cases.

Co-authored-by: Cursor <cursoragent@cursor.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 1bfc8e0. Configure here.

Comment thread recipes-devtools/python/python3-improv/imx93-jaguar-eink/onboarding-server.py Outdated
…export

Address two Cursor Bugbot findings on the advertising watchdog:

- Watchdog cancel races server stop: shutdown cancelled net_task/advert_task
  but did not await them before server.stop(), so the watchdog could be mid
  _reassert_advert() (stop/start advertising) while server.stop() tore down the
  same BlueZ/D-Bus advertisement resources. Now await the cancelled tasks so
  teardown is serialised.

- Sync D-Bus unexport blocks loop: clarify that _drop_stale_adverts()'s
  bus.unexport() is local, non-blocking bookkeeping (no awaited D-Bus round-trip
  a wedged stack could stall on) and is loop-affine so cannot be offloaded to an
  executor, which is why it is intentionally not timeout-wrapped like the
  stop/start_advertising calls. Harden the per-item guard with debug logging.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant