Rotate the INSEE password from a probed state - #383
Draft
skelz0r wants to merge 6 commits into
Draft
Conversation
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 36839055 | Triggered | Generic Password | 78b5d2f | site/spec/clients/insee_password_renewal_spec.rb | View secret |
| 36839055 | Triggered | Generic Password | 78b5d2f | site/spec/clients/insee_password_renewal_spec.rb | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secrets safely. Learn here the best practices.
- Revoke and rotate these secrets.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
skelz0r
marked this pull request as draft
September 2, 2026 12:52
skelz0r
force-pushed
the
feature/api-7345-rotation-du-mot-de-passe-insee-conception-post-incident-du
branch
4 times, most recently
from
September 7, 2026 10:24
78763f0 to
141f85f
Compare
The password was derived from the clock and pushed blindly: a single failed rotation desynchronized the application from INSEE without any way back, since repairing the password requires a valid token. The five attempts retry loop then matched INSEE's account locking threshold exactly, which is what turned that desynchronization into the outage of September 1st. INSEE's state is now probed instead of assumed. Authentication walks at most two candidates, current and previous, one attempt each, and only carries on to the next on invalid_grant: a timeout, a 5xx, a 429 or a 408 prove nothing about the password and must not spend the failure budget. Since one success resets INSEE's counter, no authentication ever exceeds one consecutive failure as long as INSEE holds one of the two candidates. Any other 4xx is a refusal of the exchange itself -- a revoked client secret, an unauthorized_client, a locked account -- and stops the walk at once: the remaining candidates cannot help, and spending them is exactly the run of consecutive failures that locks the account. Reading a rate limiting as such a refusal would turn a Retry-After of a few seconds into thirty minutes without Sirene, plus an alert about credentials that were never in question. Both candidates being refused does not conclude the walk either. The daily rotation renews the password while requests keep coming, so an authentication straddling a renewal sees the current password refused before it and the previous one refused after, and would declare a desynchronization INSEE knows nothing about. Nothing can coordinate the two, since the rotation runs in the site application and the two share no Redis, so the current password is probed one last time before concluding -- skipped when it was already the last candidate tried, where nothing can have changed since. It costs one extra refused grant on the day the account really is desynchronized, against a false alert and thirty minutes of outage on every rotation day. A single flight lock keeps the concurrent threads of an instance to one OAuth call, and a thirty minutes negative cache stops every incoming request from hammering a locked account. That cache is read a second time once the lock is held: two requests arriving together both find it empty and race for the lock, and the loser would otherwise spend its own candidates against an account the winner has just found refusing every password. Rereading the token was not enough, a failed authentication publishes no token, only the negative cache. The negative cache lives in a stable "insee" namespace rather than the default one, whose key prefix embeds the boot timestamp of the writing process: a deployment must not resume spending grants against an account just found locked. That also means Rails.cache.clear no longer wipes it between examples, hence the explicit reset in the spec helper. Beware that a per-call namespace replaces the store namespace instead of nesting under it. The lock, on the other hand, sits beside the token in the application namespace, because it only means something to the processes that can read what its holder publishes: a fleet-wide lock over a per-boot token would make every loser fail without ever being able to read the winner's token. The two applications share no Redis, so each spends its own attempt budget against the common account -- which is why each stops so early. The lock carries the identifier of the request holding it and is only released when it still matches, so an authentication outliving the TTL cannot delete the lock a successor acquired meanwhile, which would open the door to the concurrent password attempts the lock exists to prevent. The TTL covers the worst case rather than expiring in the middle of it: three attempts at ten seconds of connect plus ten of read. Acquisition tells three states apart rather than two. The cache store wraps every operation in a failsafe, so a nil return meant both "someone else holds the lock" and "Redis is gone"; reading the second as the first turned a cache blip into a total INSEE outage, each request sleeping half a second before failing on a cache that could not answer. SET NX separates them on its own, and with no cache to coordinate through the request simply authenticates. Invalidation takes the token the request actually sent and is a no-op unless the cache still holds it. Five threads carrying the same revoked token otherwise all fall through to reauthentication, and the losers delete the token the winner has just stored -- one expiry event costing as many grants as there are threads, with the single flight never converging to one authentication. Derivation guarantees one character of every class INSEE requires by overwriting a position whose class has at least two members, which always exists: sixteen characters spread over at most three present classes leave one of them with a spare. Writing on a fixed position instead would drop the sole member of a class already satisfied, and INSEE rejects such a password: the raw encoding tSu1JGtkNfTLOXRQ only lacks a special character, and writing '#' on index 3 dropped its single digit. Over 50 000 periods, 342 of them (0.7%) came out that way. Rotation leaves the request path entirely, along with the renewal interactor: it now belongs to a daily job in the site application, which shares the account and the derivation key. Derivation itself is deferred to November, so the deployment is a no-op until then.
Both applications authenticate on the same INSEE account, so they must agree on the password and on how many failures they are allowed to spend. The derivation code is duplicated identically, locked on both sides by the same known vector test: any divergence in the key handling or in the formatting now breaks the CI instead of the account. Authentication follows the same policy as siade: one attempt per candidate, carrying on only on invalid_grant, stopping at once on any other 4xx since a refused exchange cannot be helped by another password, a single flight lock released only by the request that owns it and whose TTL covers three attempts at five seconds each, and an alert naming both hypotheses, a desynchronized password or a locked account. A 429 and a 408 are the exceptions to that 4xx rule. The INSEE gateway rate limits and answers a Retry-After in seconds; treating that as a refused exchange would cost thirty minutes without Sirene and an alert on credentials that were never in question. Both stop the authentication the way a 5xx or a timeout does, without spending the next candidate or arming the guard. Classifying a response that way is a job of its own, so the OAuth call and its verdict live in INSEEOAuthExchange, leaving this class with the token cache, the lock and the failure guard. Every candidate being refused does not conclude the walk either: the daily rotation renews the password under the requests, and an authentication straddling a renewal sees the current password refused before it and the previous one refused after. Since the rotation shares no cache with the request path, the current password is probed one last time before declaring a desynchronization -- skipped when it was already the last candidate tried, where nothing can have changed since. The failure guard is read a second time once the lock is held. Two requests arriving together both find it empty and race for the lock; the loser would otherwise spend its own candidates against an account the winner has just found refusing every password. Rereading the token was not enough, a failed authentication publishes no token, only the guard. The two applications share no Redis, so nothing is pooled between them: each keeps its own token, its own lock and its own negative cache, and spends its own attempt budget against the common account. The negative cache sits in a stable "insee" namespace so that a deployment cannot resume spending grants against an account just found locked, while the lock stays beside the token, whose namespace is rebuilt at every boot -- a lock outliving what it protects would block requests that can never read the token it was taken for. Acquiring that lock tells three states apart rather than two, since the cache store returns nil both when another process holds the lock and when Redis is unreachable; reading the second as the first would turn a cache blip into a total INSEE outage. With no cache to coordinate through, the request authenticates directly. Invalidation takes the token the caller actually sent and does nothing unless the cache still holds it, so a request answering a stale 401 cannot erase a token a sibling published in the meantime. The guards are releasable from a console: they are deliberately outside the application cache prefix, where rake cache:clear cannot reach them, and an operator who has repaired the credentials should not have to wait out the TTL. The Faraday retry middleware is dropped, since it retried on UnauthorizedError and ClientError, which are exactly the answers that must never be retried. The token is cached until it expires instead of being asked again on every call.
INSEESireneAPIClient raised on a 401 but kept the bearer in cache, so every following request replayed the token INSEE had just refused, until its announced expiration around seven days later. A password renewal or a token revocation therefore broke the organisation payload jobs for a week. A 401 now drops the cached token and replays the request once, which reauthenticates through the candidate list. The client remembers the bearer it actually sent and hands it to the invalidation, so a request answering a stale 401 cannot erase a token another request refreshed between its own read and its rescue. siade already did this in INSEE::MakeRequest.
siade has neither a job framework nor a writable database, site has GoodJob with cron. Since both share the INSEE account and the derivation key, site rotates the password for everyone. INSEE::PasswordRotation holds the whole state machine: converge on the current password, knowing INSEE holds either it or one fallback. Probe the target first -- on a normal day it is granted and there is nothing to do -- then the fallback, and renew from it. The job passes the previous period's password as the fallback; an operator leaving the bypass passes the bypass value. Expressing both as one operation is what keeps them honest: a task duplicating this machine would drift from it, and would inherit none of its guards. The job is stateless and idempotent. Replayed the next day it converges, so a failed rotation is retried daily and alerts every time until a human steps in, well within the twenty nine days of margin left by the ninety days validity. It costs one successful authentication on a normal day, one failure plus one success plus one renewal on a rotation day, and never more than two failures. Finding both passwords rejected does not merely alert: it arms the same guard the request path arms, so the applications stop spending candidates against an account just found desynchronized instead of waiting for a request to discover it again. Every provider failure is reported for what it is. An unavailable probe proves nothing about the password it probed, on the previous one as on the current one, so both take the warn and wait path rather than the desynchronization alert. A renewal that never reaches INSEE is reported like any other renewal failure instead of raising out of the job: ApplicationJob defines no retry policy, so the exception would lose the alert meant to fire on a failed rotation and leave the next daily run as the only recourse. It runs at 00:05 rather than a few minutes before midnight: rotating early would leave INSEE on a password no application lists among its candidates, hence a guaranteed double failure on any fresh authentication. Four independent guards keep it to a single execution: production only in the schedule, the FRONTAL environment variable as HealthcheckJob does, a GoodJob concurrency key shared by every machine, and the idempotence of the job itself. Leaving the bypass runs the same code from a console. Going back to derivation used to mean a window during which the applications authenticate with a password INSEE no longer holds; the operation renews from the bypass value to the derived one while both applications still list the bypass first and the derived password second, so they keep authenticating throughout, and the bypass credentials are only removed afterwards. Replaying it is safe: a timeout may have gone through on INSEE's side, in which case the bypass no longer authenticates while the derived password does, which the first probe detects without calling the renewal endpoint a second time. It refuses to run before the derivation window opens, where the current password is still the static one and the operation would hand the account back to the very value the bypass exists to work around.
Authentication used to let Faraday errors bubble out of #access_token, and UpdateOrganizationINSEEPayloadJob retried them forever. Wrapping the OAuth call in an attempt that rescues Faraday::Error and re-raises TemporaryError silently took that away: the job knows nothing about that class, GoodJob does not retry what a job does not declare, so a single 5xx or timeout on the INSEE token endpoint now loses the payload. Nothing enqueues that job again either, it only runs on a DataPass webhook, so the organization would keep an empty insee_payload until the next habilitation touches it. Attempts are bounded, unlike the Faraday ones: a temporary error also covers the thirty minutes of negative cache and the single flight waiter, where retrying forever would only add noise to an outage that is already reported.
ProviderAuthenticationError had a single emitter, INSEE::RenewPassword, whose errors rotate_password_if_needed! discarded -- it only read success? -- so subcode 01006 was unreachable and documented nowhere. Authenticate now emits it, and MakeRequest concatenates its errors into the organizer context, so it surfaces as a 502 on the fifteen Sirene endpoints. SDK users would have met an undocumented code carrying no retry semantics, where the temporary error they know does carry them. Declaring it on the interactor rather than in the baseline list keeps it on the INSEE endpoints alone. Both SDKs were regenerated and are unchanged: they model errors by HTTP status and pass the code through, without a table to keep in step.
skelz0r
force-pushed
the
feature/api-7345-rotation-du-mot-de-passe-insee-conception-post-incident-du
branch
from
September 7, 2026 15:24
1b6bb51 to
3e378ad
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://linear.app/pole-api/issue/API-7345
The September 1st outage came from a password derived from the clock and
pushed blindly: one failed rotation desynchronized us from INSEE with no way
back, since repairing the password requires a valid token, and the five
attempts retry loop matched INSEE's account locking threshold exactly.
INSEE's state is no longer guessed nor memoized, it is probed. Authentication
walks at most two candidates, current and previous, one attempt each, and only
falls back on
invalid_grant: a timeout or a 5xx proves nothing about thepassword and must not spend the failure budget. A single flight lock keeps the
concurrent threads of an instance to one OAuth call, and a thirty minutes
negative cache stops every incoming request from hammering a locked account.
Since one success resets INSEE's counter, no authentication ever exceeds one
consecutive failure as long as INSEE holds one of the two candidates. Both
applications share the same account, so they now share the same policy and the
same derivation code, locked on both sides by an identical known vector test.
Rotation leaves the request path entirely and becomes a daily GoodJob cron in
site production, the only place with a job framework and a writable database.
It is stateless and idempotent: it probes the current password and only renews
when INSEE still holds the previous one, so a failed rotation is retried every
day and alerts every time, well within the twenty nine days of margin left by
the ninety days validity. Four independent guards keep it to a single
execution: production only in the schedule,
FRONTALasHealthcheckJobdoes,a GoodJob concurrency key, and the idempotence of the job itself. A
insee:rotate_from_bypasstask hands the account back to derivation without aservice window, and is safe to replay.
Derivation itself starts on November 1st: until then every candidate list
collapses to the static credential, so this deployment changes no password and
costs no failed authentication.