Skip to content

Stop sending admin emails on every HTTP 500 in production#1356

Merged
mihow merged 2 commits into
mainfrom
fix/disable-prod-admin-email-handler
Jul 23, 2026
Merged

Stop sending admin emails on every HTTP 500 in production#1356
mihow merged 2 commits into
mainfrom
fix/disable-prod-admin-email-handler

Conversation

@mihow

@mihow mihow commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

In production, every uncaught HTTP 500 triggers Django's default admin-error email, sent synchronously inside the request — one outbound email per error, each holding a worker thread for the duration of the send. During an error storm this becomes an email storm that ties up the worker pool and makes an incident worse. Sentry already captures every error independently, so these emails add no observability. This change disables that handler by default in deployed environments and puts it behind an environment variable, so it can be re-enabled per environment if a non-blocking (async/queued) handler is configured later.

List of Changes

  1. Admin error emails are no longer sent on production 500s by default. Operator-facing effect: an error spike can no longer snowball into a synchronous outbound-email storm that starves the worker pool. (Implementation: gate Django's default mail_admins AdminEmailHandler behind DJANGO_ADMIN_ERROR_EMAILS, default False; when off, the django logger routes to console only at INFO so 4xx/5xx request lines still reach stdout, and no longer propagates to the handler.)
  2. Configurable per environment: set DJANGO_ADMIN_ERROR_EMAILS=True to re-enable admin error email where wanted. Covers production, demo, and staging (same settings module).

Notes

  • Sentry error capture is unaffected — it hooks the logging system via LoggingIntegration, independent of the LOGGING handlers list.
  • Surfaced during a production incident where the synchronous admin-email path amplified a 500 storm.

Summary by CodeRabbit

  • Bug Fixes
    • Updated production error logging behavior to better control when Django errors are sent to admin email alerts.
    • Added a new production toggle to route Django logs to the console at info level when admin error emails are disabled, helping keep logging behavior consistent.

Django's DEFAULT_LOGGING routes the "django" logger to AdminEmailHandler
("mail_admins"), which sends a synchronous outbound email for each HTTP 500.
Override it to console-only so the AdminEmailHandler is bypassed. Sentry
already captures all ERROR-level records independently via LoggingIntegration,
so no observability is lost.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings June 30, 2026 01:39
@netlify

netlify Bot commented Jun 30, 2026

Copy link
Copy Markdown

Deploy Preview for antenna-ssec ready!

Name Link
🔨 Latest commit 8ccbc17
🔍 Latest deploy log https://app.netlify.com/projects/antenna-ssec/deploys/6a6188179e83ff0008353b9e
😎 Deploy Preview https://deploy-preview-1356--antenna-ssec.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@netlify

netlify Bot commented Jun 30, 2026

Copy link
Copy Markdown

Deploy Preview for antenna-preview ready!

Name Link
🔨 Latest commit 8ccbc17
🔍 Latest deploy log https://app.netlify.com/projects/antenna-preview/deploys/6a618817bdb9d60009eba287
😎 Deploy Preview https://deploy-preview-1356--antenna-preview.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
Lighthouse
Lighthouse
1 paths audited
Performance: 55 (🔴 down 10 from production)
Accessibility: 81 (🔴 down 8 from production)
Best Practices: 92 (🔴 down 8 from production)
SEO: 92 (no change from production)
PWA: 80 (no change from production)
View the detailed breakdown and full score reports
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai

coderabbitai Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

@mihow, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 58 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: dcb988aa-845e-4434-a3ab-fb0bc13ccea1

📥 Commits

Reviewing files that changed from the base of the PR and between 585b1b0 and 8ccbc17.

📒 Files selected for processing (1)
  • config/settings/production.py
📝 Walkthrough

Walkthrough

Adds ADMIN_ERROR_EMAILS env bool (DJANGO_ADMIN_ERROR_EMAILS, default False) to production.py. When False, the LOGGING["loggers"]["django"] entry is overridden to emit at INFO level to the console handler with propagate=False, suppressing admin error emails.

Changes

Admin Error Email Logging Toggle

Layer / File(s) Summary
Conditional django logger override
config/settings/production.py
Adds ADMIN_ERROR_EMAILS env flag; when False, overrides the django logger to INFO/console/propagate=False, bypassing admin email notifications.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Poem

A toggle appears, small but keen,
To hush the emails, keep logs clean,
When False is set, the django speaks
To console only, no admin squeaks,
The rabbit hops with logging peace! 🐇

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly matches the main change: disabling production admin error emails for HTTP 500s.
Description check ✅ Passed The description covers the summary, changes, and operational notes, but omits several template sections like testing, deployment notes, and checklist.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/disable-prod-admin-email-handler

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adjusts production Django logging configuration to prevent synchronous admin-error emails on HTTP 500s (intended to avoid email storms during incidents), while keeping error visibility via Sentry and stdout logging.

Changes:

  • Introduces DJANGO_ADMIN_ERROR_EMAILS env var (default False) to control admin-error emailing behavior.
  • When disabled, modifies production LOGGING to route Django logs to console and prevent propagation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread config/settings/production.py

@mihow mihow left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude says: (takeaway review, drafted by Claude on Michael's behalf.)

The fix is correct and safe to ship. I traced it in a running container rather than from the diff alone, and the mechanism holds up:

  • The admin error emails aren't wired anywhere in the repo — they come from Django's DEFAULT_LOGGING, which configure_logging() applies before this module's LOGGING. Both use disable_existing_loggers: False, so the django logger silently keeps its default mail_admins handler unless something redefines it. This PR redefines it, which is exactly the right lever, and explains why grepping for mail_admins finds nothing while the emails still send.
  • No loss of error visibility. django.request (where unhandled 500s are logged) has no handlers of its own and propagates up into the django logger, so it still reaches the console. Sentry is entirely unaffected: LoggingIntegration patches logging.Logger.callHandlers upstream of handler dispatch, so it captures the error regardless of the handler list. Console logging is preserved, and there's no double-logging.

One structural point worth addressing before merge (inline suggestion below): the "on" path is implicit. When DJANGO_ADMIN_ERROR_EMAILS=True, the code does nothing and inherits mail_admins from Django's internals — so "on" works only as long as Django's DEFAULT_LOGGING keeps attaching that handler, and nobody reading production.py can see where the emails come from. Making it an explicit branch costs a few lines.

Nothing else needs changing — the fix does what it says.

Comment thread config/settings/production.py
When DJANGO_ADMIN_ERROR_EMAILS=True the mail_admins handler comes from
Django's DEFAULT_LOGGING, not this file. An explicit else branch makes
that visible at the site of the toggle instead of hidden in Django's
defaults.

Co-Authored-By: Claude <noreply@anthropic.com>
@mihow
mihow merged commit 13de599 into main Jul 23, 2026
7 checks passed
@mihow
mihow deleted the fix/disable-prod-admin-email-handler branch July 23, 2026 03:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants