Cancel DB queries when the 8Knot instance shuts down

[EngCaioFonseca]

Pull Request Change Description

There seems to have 4 factors contributing to this issue:

Closes #802

Long-running queries against the Augur database kept running after the 8Knot instance was shut down. Four things combined to cause this:

No statement_timeout on any Augur connection, so Postgres had no reason to ever stop.
Postgres does not check whether the client is still there while it is busy inside a query — it only finds out on the next write to the socket. A container killed mid-SELECT left the server burning CPU indefinitely.
with psycopg2.connect(...) does not close the connection; it only ends the transaction. A worker killed between fetches left its session idle in transaction, holding the server-side cursor open.
Celery's --soft-time-limit cannot interrupt a blocking psycopg2 execute(), so the limits we already had, bounded the worker, not the query. That is what produced the orphans in the first place.

The fix is on the server-side, where no client-side handler gets to run.

Every connection to Augur now carries statement_timeout, idle_in_transaction_session_timeout, and TCP keepalives (set on the server's end of the socket). On PostgreSQL 14+ we also enable client_connection_check_interval, which makes Postgres poll for a vanished client during a query and abort promptly. That setting also requires POLLRDHUP, which is Linux-only, so it is applied best-effort — hosts that reject it log a warning and stay bounded by the timeouts.

Two supporting changes:

Query tasks no longer retry on QueryCanceled. Adding statement_timeout means a slow query now fails instead of hanging, and the existing autoretry_for=(Exception,) with max_retries: 5 would have retried it five times, each burning another 500s of Augur DB time. A dont_autoretry_for on the base task class prevents that.
caching_wrapper re-raises the original exception instead of flattening it into a generic Exception, which is what lets the above actually match.

Three tunables were added to .env.sample, all in milliseconds: AUGUR_STATEMENT_TIMEOUT_MS (500s, worker queries — kept under Celery's 540s soft limit so the DB aborts first), AUGUR_ENGINE_STATEMENT_TIMEOUT_MS (30min, the app-server's searchbar query, which is legitimately much slower), and AUGUR_IDLE_TX_TIMEOUT_MS (2min).

Generative AI disclosure

Please select one option:

• This contribution was NOT assisted or created by Generative AI tools.
• This contribution was assisted or created by Generative AI tools.

If AI tools were used, please provide details below:
- What tools were used? Claude Opus 5, GPT 5.6 Sol
- How were these tools used? Investigation, initial draft, review.
- Did you review these outputs before submitting this PR? Yes, reviewed it.