Skip to content

Add healthchecks.io dead-man's-switch to paperless backup - #102

Open
michaelbarton wants to merge 1 commit into
masterfrom
add/paperless-backup-healthcheck
Open

Add healthchecks.io dead-man's-switch to paperless backup#102
michaelbarton wants to merge 1 commit into
masterfrom
add/paperless-backup-healthcheck

Conversation

@michaelbarton

Copy link
Copy Markdown
Owner

Summary

  • We just found the daily paperless backup LaunchAgent was never loaded on this machine — the job silently didn't run for 62 days with nothing to flag it, since there was no failed run to notice, just no run at all.
  • backup.sh now pings a healthchecks.io check on success, and its /fail endpoint (via an ERR trap) if any step fails.
  • If the scheduled ping doesn't arrive (crashed run, unloaded LaunchAgent, machine off, etc.), healthchecks.io raises the alert externally — this is what actually catches the "job never ran" failure mode, which a local "notify on script error" never would have.
  • The ping URL is optional, read from Keychain (paperless-backup-healthcheck-url); if unset, backup.sh just skips the ping and behaves exactly as before.

Test plan

  • bash -n backup.sh — syntax check passes.
  • Isolated the trap + ping logic and verified: a failing step triggers the ERR trap, logs the failure, and best-effort pings /fail without the ping itself being able to crash the script (|| true).
  • Verified the success path calls ping_healthcheck after "Paperless backup complete" and exits 0.

Follow-up (manual, one-time)

Documented in ansible/tasks/paperless_backup.yml: create a free check at healthchecks.io with a 24h period (~a few hours grace) matching the daily 3am schedule, then:

security add-generic-password -a $USER -s paperless-backup-healthcheck-url -w 'https://hc-ping.com/<uuid>'

🤖 Generated with Claude Code

We just found the daily backup LaunchAgent was never actually loaded
on this machine, so the job silently didn't run for 62 days with no
error to notice — a plain "notify on script failure" wouldn't have
caught this since the script never even started.

backup.sh now pings a healthchecks.io check on success, and its
/fail endpoint via an ERR trap if any step fails. If the ping doesn't
arrive on schedule (crashed run, unloaded agent, machine off),
healthchecks.io raises the alert externally instead of relying on
something local to notice its own absence. The ping URL is optional
(read from Keychain) so backup.sh still runs fine without it configured.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@michaelbarton
michaelbarton force-pushed the add/paperless-backup-healthcheck branch from 0abf99e to c8cfae4 Compare August 10, 2026 22:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant