Repository navigation
feat(cloudformation): refuse updates after a failed rollback and add ContinueUpdateRollback - #4949
Conversation
…a failed rollback and add ContinueUpdateRollback
…a failed rollback and add ContinueUpdateRollback
…a failed rollback and add ContinueUpdateRollback
…a failed rollback and add ContinueUpdateRollback
…a failed rollback and add ContinueUpdateRollback
cd18ec0 to
3d24fbc
Compare
…a failed rollback and add ContinueUpdateRollback
3d24fbc to
3109d12
Compare
…a failed rollback and add ContinueUpdateRollback
3109d12 to
50e1bc6
Compare
…a failed rollback and add ContinueUpdateRollback
3e14ed6 to
3ef0ae8
Compare
PR Summary by QodoHandle failed CloudFormation update rollbacks and add rollback retries
AI Description
Diagram
High-Level Assessment
Files changed (15)
|
Code Review by Qodo
1.
|
There was a problem hiding this comment.
🟡 Changes recommended
The new Query action does not synchronously validate required and constrained AWS request parameters.
1 open finding
What changed in this PR
Adds AWS-compatible recovery for CloudFormation stacks in UPDATE_ROLLBACK_FAILED.
Changes:
- Implements
ContinueUpdateRollback, resource skipping, and persisted rollback snapshots. - Refuses updates and update change sets while rollback recovery is required.
- Adds protocol responses, documentation, and comprehensive rollback tests.
| File | Description |
|---|---|
ReplacementCleanupTest.java |
Tests replacement-delete retries. |
ElbV2CfnProvisionerTest.java |
Updates ELB retry expectations. |
EcsCfnProvisionerTest.java |
Updates ECS retry expectations. |
CognitoResourceServerCfnIntegrationTest.java |
Uses rollback recovery before later updates. |
CloudFormationServiceRollbackTest.java |
Tests snapshots, skips, persistence, and retries. |
CloudFormationPipesCleanupIntegrationTest.java |
Adds recovery before pipe updates. |
CloudFormationContinueUpdateRollbackIntegrationTest.java |
Covers the new action end to end. |
ReplacementCleanup.java |
Tracks and retries undeleted replacements. |
StackUpdateSnapshot.java |
Defines persisted pre-update state. |
Stack.java |
Stores pending rollback metadata. |
CloudFormationService.java |
Implements refusal and rollback continuation logic. |
CloudFormationQueryHandler.java |
Adds the Query action and status reason response. |
AwsQueryController.java |
Registers the new CloudFormation action. |
docs/services/index.md |
Updates the operation count. |
docs/services/cloudformation.md |
Documents failed rollback recovery. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
3ef0ae8 to
73214f5
Compare
73214f5 to
a148188
Compare
pgermosen
left a comment
There was a problem hiding this comment.
@mariotoffia really thorough work, and the AWS measurements made it easy to check.
The Cognito resource server paragraph in the docs still describes the old flow, though. It points a failed restoration at a later update, and says an update can stay stuck in cleanup. Now that updates are refused after a failed rollback, that retry is ContinueUpdateRollback. A skipped server comes out UPDATE_COMPLETE, so the next update just drops its pending restoration and the stuck-cleanup case can't happen. You already turned its test into the refusal test, so that sentence should go too.
In finishUpdateRollback the failed branch still sets the status before the reason and the event. Please flip it to match the order you used everywhere else.
a148188 to
66340cd
Compare
|
@pgermosen both addressed in 66340cd (rebased on current main):
CloudFormation package: 3201 tests, 0 failures. Checkstyle, |
66340cd to
21d4986
Compare
…ContinueUpdateRollback
21d4986 to
b011b00
Compare
pgermosen
left a comment
There was a problem hiding this comment.
@mariotoffia thanks, both are in: the resource-server docs now point a failed restoration at ContinueUpdateRollback without the stuck-cleanup sentence, and the failed branch of finishUpdateRollback sets the status last. Recording only the walker's failures before finishing is a good catch too.
…nueUpdateRollback in schedules floci-io#4949 refuses an update over UPDATE_ROLLBACK_FAILED until ContinueUpdateRollback finishes the rollback, as AWS does. floci-io#4834, merged minutes earlier, had seven tests that update over that status, and main has failed them since c4a809f. Six are in CloudFormationSchedulerScheduleIntegrationTest. They now reach their next update the way AWS requires: ContinueUpdateRollback, which retries the schedule's own rollback, or, where the replacement's delete has to keep failing, a skip of the schedule. The two whose update was meant to be blocked by the pending rollback now check that the update is refused. The seventh shows a real clash. floci-io#4949 made a second ReplacementCleanup.rollback retry the delete the first one could not make, which is ContinueUpdateRollback's job. The Scheduler provisioner also calls rollbackUpdate inside an update, to undo its own failed call, so after a skip a failed group move deleted the earlier orphan there and then, and a failure of that delete would have failed the move's restore. A schedule update now ends that retry first, through ReplacementCleanup.endRollbackRetry, so the orphan stays owed for the next committed update or the stack delete, as both changes document.
…nueUpdateRollback floci-io#4949 refuses an update over UPDATE_ROLLBACK_FAILED until ContinueUpdateRollback finishes the rollback, as AWS does. floci-io#4834, merged minutes earlier, had seven tests that update over that status, and main has failed them since c4a809f. Six are in CloudFormationSchedulerScheduleIntegrationTest. They now reach their next update the way AWS requires: ContinueUpdateRollback, which retries the schedule's own rollback, or, where the replacement's delete has to keep failing, a skip of the schedule. The two whose update was meant to be blocked by the pending rollback now check that the update is refused. The seventh shows a real clash. floci-io#4949 made a second ReplacementCleanup.rollback retry the delete the first one could not make, which is ContinueUpdateRollback's job. The Scheduler provisioner also calls rollbackUpdate inside an update, to undo its own failed call, so after a skip a failed group move deleted the earlier orphan there and then, and a failure of that delete would have failed the move's restore. An update now ends that retry as it starts, through ReplacementCleanup.endRollbackRetry, so the orphan stays owed for the next committed update or the stack delete, as both changes document. Three more provisioners roll an update back before ReplacementCleanup.record has cleared the retry, and each returns as soon as the retry ran, skipping its own restore. CloudWatchDashboard and IotAuthorizer unwind a failed in-place update that way, and CognitoResourceServer's failed attempt is rolled back the same way, so after a skip the update's snapshot was never put back while the resource was reported restored. They end the retry as an update starts too.
* fix(ses): delete by id and filter by source in /_aws/ses The inspection endpoint follows LocalStack's contract, as docs/services/ses.md says, but differed from it in two ways. LocalStack documents that "The query parameter id can be used to delete only a specific message", and that "Query parameters id and email can be used to filter by message ID and message source respectively". - DELETE /_aws/ses?id=<id> cleared every message. It now deletes only that message, in whichever region it was sent. An unknown id deletes nothing. - GET /_aws/ses?email=<address> matched the address against each message's ToAddresses. It now matches the message's Source, as LocalStack does. GET also listed messages in storage hash order. It now lists them oldest first by Timestamp, the order LocalStack keeps them in. Floci's own tests read Cognito codes and other mail with ?email=<recipient>. They now list the mailbox and pick the recipient's messages out of it, through a new SesMailbox test helper in src/test and the same filtering in the Java and Node compatibility tests. A raw send's mailbox entry shows RawData without its recipients, so the test for recipients read from a raw message's headers checks the stored record. * fix(codeartifact): run the pypiserver sidecar under an init process so a stop ends it at once The pypiserver image's entrypoint execs pypi-server, which installs no SIGTERM handler. The kernel gives PID 1 no default signal actions, so the container ignored every stop request and went only when Docker sent SIGKILL at the end of the 5-second timeout: each reset, repository delete and shutdown waited that long for every pypi repository's container, one after another. ContainerBuilder gains withInit(), carried on ContainerSpec as init and passed to Docker as HostConfig.Init (docker run --init), and PypiserverSidecarManager uses it. Docker's init runs as PID 1 and forwards the SIGTERM, so a stop now returns in a fraction of a second. Verdaccio already handles the signal itself and is unchanged. * fix(firehose): check buffers every second so delivery follows the buffering interval AWS delivers a Firehose buffer once the first buffer condition is met. Floci checked for elapsed buffering intervals on a 10-second tick, so a buffer was delivered up to 10 seconds after its IntervalInSeconds ended: a 5-second interval landed anywhere from 5 to 15 seconds after the first record. The default tick is now one second; the check is a scan of the in-memory buffers. FirehoseFlushIntegrationTest's profile only set the same one-second tick, so it is removed and the class shares the default test application. The compat tests that wait for a delivery checked S3 every 2 or 5 seconds; against Floci they now check every 250 ms, and keep their cadence against real AWS. * test: make subclass mocks the default so mocking stays cheap in a loaded test JVM Mockito's default inline mock maker redefines every class it mocks in the running JVM. A redefinition gets slow once the test JVM has loaded the Quarkus applications of the classes that ran before, which in CI is most of the run: a unit class's first test spent seconds building its mocks, and EmulatorLifecycleTest, which mocks some fifty types, spent 22 seconds. Subclass mocks generate a subclass instead and redefine nothing. After twelve Quarkus applications in one JVM, the fourteen Mockito classes whose first test was slowest in CI take 20.5 seconds instead of 41.9, and their first tests 6.9 seconds instead of 27.8. mockito-extensions/org.mockito.plugins.MockMaker makes the subclass mock maker the default. A subclass cannot extend a final class, record or enum, so those mocks go through InlineMocks.inlineMock and inlineSpy, or @mock(mockMaker = MockMakers.INLINE), and EmulatorLifecycleTest's static mock of Quarkus asks for the inline maker. The ContainerSpec mocks a mocked ContainerBuilder.Builder returned were opaque values, so they are real specs now. * fix(cloudformation): leave the last rollback's failed delete to ContinueUpdateRollback #4949 refuses an update over UPDATE_ROLLBACK_FAILED until ContinueUpdateRollback finishes the rollback, as AWS does. #4834, merged minutes earlier, had seven tests that update over that status, and main has failed them since c4a809f. Six are in CloudFormationSchedulerScheduleIntegrationTest. They now reach their next update the way AWS requires: ContinueUpdateRollback, which retries the schedule's own rollback, or, where the replacement's delete has to keep failing, a skip of the schedule. The two whose update was meant to be blocked by the pending rollback now check that the update is refused. The seventh shows a real clash. #4949 made a second ReplacementCleanup.rollback retry the delete the first one could not make, which is ContinueUpdateRollback's job. The Scheduler provisioner also calls rollbackUpdate inside an update, to undo its own failed call, so after a skip a failed group move deleted the earlier orphan there and then, and a failure of that delete would have failed the move's restore. An update now ends that retry as it starts, through ReplacementCleanup.endRollbackRetry, so the orphan stays owed for the next committed update or the stack delete, as both changes document. Three more provisioners roll an update back before ReplacementCleanup.record has cleared the retry, and each returns as soon as the retry ran, skipping its own restore. CloudWatchDashboard and IotAuthorizer unwind a failed in-place update that way, and CognitoResourceServer's failed attempt is rolled back the same way, so after a skip the update's snapshot was never put back while the resource was reported restored. They end the retry as an update starts too. --------- Co-authored-by: Priamo <pgermosen@gmail.com>
|
🎉 This PR is included in version 2.3.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |



Summary
Closes #4945
When an update fails and the rollback of one of its resources fails too, the stack settles in
UPDATE_ROLLBACK_FAILED. Floci now handles that state as AWS does.Refused updates.
UpdateStackand anUPDATEchange set are refused withValidationErrorStack:<arn> is in UPDATE_ROLLBACK_FAILED state and can not be updated.Floci used to accept them and build the new update on resources whose state was unknown.DeleteStackstays allowed.ContinueUpdateRollback. The new action takesStackName,ResourcesToSkipandForceRollback:UPDATE_ROLLBACK_IN_PROGRESSevent with the reasonUser Initiated.ResourcesToSkipis reportedUPDATE_COMPLETEwith the reasonResource skipped during UpdateRollback, and is not rolled back.ForceRollback=trueskips everyUPDATE_FAILEDresource of the stack the same way. When it skipped one, the completed stack's reason says the skipped resources may have drifted, with AWS's text.UPDATE_ROLLBACK_COMPLETE. It gets back the template, parameters and outputs from before the failed update, and takes updates again.UPDATE_ROLLBACK_FAILED, and a later call can retry it or skip it. An unexpected error during the retry settles the stack there too, so it is never left inUPDATE_ROLLBACK_IN_PROGRESS.To make that possible, a failed rollback now keeps on the stack:
StackUpdateSnapshot, which used to be a private recordBoth persist, so a retry works after a restart too. A retry reads them in
UPDATE_ROLLBACK_IN_PROGRESS, and they are cleared once a rollback completes or the stack is deleted.As on AWS:
ValidationErrorbefore the stack is looked up.StackNamemust be present and match[a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]*, and every skip must match[a-zA-Z0-9]+|[a-zA-Z][-a-zA-Z0-9]*\.[a-zA-Z0-9]+. Every violation is reported in one message,StackNamefirst.ForceRollback=truewith skips is refused withValidationErrorForceRollback and ResourcesToSkip are mutually exclusive, and a missing stack withStack: <StackName> does not exist.UPDATE_ROLLBACK_IN_PROGRESS, withValidationErrorRollbackUpdatedStack cannot be called from current stack status.ResourcesToSkipis checked after the call is accepted. A bad skip returns the stack toUPDATE_ROLLBACK_FAILEDwith AWS's reason and touches no resource.DescribeStackResourcenow returnsResourceStatusReason. That is where the skip reason shows.What is covered
Legend: ✅ full, 🟡 partial, ❌ not implemented
UpdateStackUPDATE_ROLLBACK_FAILEDValidationErrorwith AWS's textCreateChangeSetUPDATEchange set on a stack inUPDATE_ROLLBACK_FAILEDValidationErrorDeleteStackUPDATE_ROLLBACK_FAILEDContinueUpdateRollbackUPDATE_ROLLBACK_COMPLETE, orUPDATE_ROLLBACK_FAILEDagainContinueUpdateRollbackResourcesToSkipofUPDATE_FAILEDresourcesUPDATE_COMPLETE"Resource skipped during UpdateRollback"; the skipped resource keeps the state the failed update left it inContinueUpdateRollbackForceRollbackUPDATE_FAILEDresource is skipped; the stack reason names the possible drift. Refused together withResourcesToSkipContinueUpdateRollbackStackNameorResourcesToSkipValidationErrorbefore the stack lookup, every violation in one messageContinueUpdateRollbackValidationErrorStack: <StackName> does not existContinueUpdateRollbackValidationError"RollbackUpdatedStack cannot be called from current stack status"ContinueUpdateRollbackUPDATE_FAILEDUPDATE_ROLLBACK_FAILEDwith AWS's reason: "Resources with logicalIds [X] do not belong to stack S" or "Only the resources in UPDATE_FAILED state are allowed to be skipped"ContinueUpdateRollbackUPDATE_ROLLBACK_FAILED"Stack [X] does not exist"ContinueUpdateRollbackNested.Child)ContinueUpdateRollbackUPDATE_ROLLBACK_COMPLETEonce it is goneContinueUpdateRollbackContinueUpdateRollbackRoleARN,ClientRequestTokenContinueUpdateRollbackUPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESSContinueUpdateRollbackUPDATE_FAILEDresource and keeps the template it holdsContinueUpdateRollbackDeleteStackracing the retryContinueUpdateRollbackContinueUpdateRollbackCREATE_COMPLETEfrom the walker; AWS reportsUPDATE_COMPLETEFiles
CloudFormationService:continueUpdateRollback: validates the request, then checks the status and moves toUPDATE_ROLLBACK_IN_PROGRESSin one step under the stack's lock, so a racing update or second retry is refused. The rest runs on the operation executor.validateContinueUpdateRollback: theStackNameandResourcesToSkipconstraints, in AWS's message format.resumeUpdateRollback: checks the skips (allUPDATE_FAILEDresources when forced), reports the skipped resources, and re-runs the rollback walker over the owed resources. It records what is still owed before the walker and again after it, so a later failure never rolls back the same resource twice. An unexpected error settles the stack inUPDATE_ROLLBACK_FAILED.invalidSkipReason: why AWS would fail the rollback over the skips.finishUpdateRollback: shared by the first rollback and each retry. It keeps or clears the snapshot and the owed ids.UPDATE_ROLLBACK_FAILED.model/StackUpdateSnapshot: the record moved out of the service, registered for reflection.model/Stack:updateRollbackSnapshotandupdateRollbackResourceIds.CloudFormationQueryHandler: theContinueUpdateRollbackaction, includingForceRollback, with an empty<ContinueUpdateRollbackResult/>, andResourceStatusReasoninDescribeStackResource.AwsQueryController:ContinueUpdateRollbackin the CloudFormation action list.provisioners/ReplacementCleanup: a rollback whose delete of the replacement fails now names it on the record, so a retry deletes it again instead of reporting that the type cannot roll back. A later provision drops the name; the replacement stays owed to the next committed cleanup or the stack delete, as before.docs/services/cloudformation.md: the action row and a "Failed update rollbacks" section.docs/services/index.md: the CloudFormation operation count, 35 to 36.Tests:
CloudFormationContinueUpdateRollbackIntegrationTest(new, 14 tests):DeleteStackworksForceRollbackskips the failed resources, sets the drift reason and unblocks updatesStackName, an empty one (length and pattern), one outside the pattern, a bad skip rejecting the whole list, and a missingStackNamewith a bad skip reported togetherForceRollbackwith skips refused without touching the stackCloudFormationServiceRollbackTest, 8 new tests:UPDATE_ROLLBACK_FAILED, and a later call can continue itUPDATE_FAILEDresource and sets the drift reasonReplacementCleanupTest, 3 new tests: a retry deletes the replacement the failed rollback left, a retry that fails again keeps it owed, and a later provision stops the retry while keeping it owed.UPDATE_ROLLBACK_FAILEDnow continue the rollback first, since that update is refused:CloudFormationPipesCleanupIntegrationTest: continues the rollback, skipping the pipe.CognitoResourceServerCfnIntegrationTest(from feat(cloudformation): provision Cognito resource servers #4832), 6 tests: a plain retry where the restore can be redone, a skip ofServerwhere the test needs the failed state kept, and one test now asserting the refusal.EcsCfnProvisionerTestandElbV2CfnProvisionerTest, one test each: a second rollback asserted it deleted nothing; it now retries the failed delete.Type of change
fix:)feat:)feat!:orfix!:)AWS Compatibility
Measured with AWS CLI v2 2.37.7 in eu-west-1 on 2026-10-02. A stack held a custom resource whose rollback failed until told otherwise, and a bucket the update added with an invalid name.
update-stackandcreate-change-set --change-set-type UPDATEon theUPDATE_ROLLBACK_FAILEDstack failed withValidationErrorStack:<arn> is in UPDATE_ROLLBACK_FAILED state and can not be updated.list-change-setsstayed empty.continue-update-rollbackon a stack inUPDATE_COMPLETE,CREATE_COMPLETE,UPDATE_ROLLBACK_COMPLETEorUPDATE_ROLLBACK_IN_PROGRESSfailed withValidationError"RollbackUpdatedStack cannot be called from current stack status".<ContinueUpdateRollbackResult/>, then anUPDATE_ROLLBACK_IN_PROGRESSstack event "User Initiated".UPDATE_ROLLBACK_FAILED.--resources-to-skipwith the resource gaveUPDATE_COMPLETE"Resource skipped during UpdateRollback" without calling the handler. The stack then endedUPDATE_ROLLBACK_COMPLETE, anddescribe-stack-resourceshowed that status and reason.UPDATE_FAILED, not in the stack, or of a missing nested stack were accepted with 200. The stack then returned toUPDATE_ROLLBACK_FAILEDwith the reasons listed in the table above.update-stackwas accepted and updated from the template from before the failed update.AWS also passed through
UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESSto delete the added bucket. Floci deletes it during the rollback, as it already did.The same scenario ran with AWS CLI v2 2.37.7 against this branch. It used a log group whose rollback fails, because Floci has no failing custom resource handler.
wait stack-update-completeafter the failing updateupdate-stackValidationErrorwith AWS's textcreate-change-set --change-set-type UPDATEValidationErrorwith AWS's textcontinue-update-rollbackUPDATE_ROLLBACK_FAILEDagaincontinue-update-rollback --resources-to-skip LogGroupwait stack-rollback-completerc 0.UPDATE_ROLLBACK_COMPLETE, and the resource showsUPDATE_COMPLETE"Resource skipped during UpdateRollback"continue-update-rollbackonUPDATE_ROLLBACK_COMPLETEValidationError"RollbackUpdatedStack cannot be called from current stack status"update-stack, thenwait stack-update-completeMeasured again with AWS CLI v2 2.37.13 in eu-west-1 on 2026-10-13, for the request checks and
ForceRollback:StackName, and a skip outside its pattern, failed with HTTP 400ValidationErrorbefore the stack lookup. All violations came in one message,StackNamefirst. A 300-character skip was accepted: a skip has no length limit.--force-rollbackwith--resources-to-skipfailed withValidationError"ForceRollback and ResourcesToSkip are mutually exclusive" and left the stack unchanged.continue-update-rollbackon a stack that does not exist failed withValidationError"Stack: does not exist".--force-rollbackreported the failed resourceUPDATE_COMPLETE"Resource skipped during UpdateRollback" without calling its handler. The stack endedUPDATE_ROLLBACK_COMPLETEwith the pre-update template, and its reason, also on the completion event, was "One or more resources were skipped during UpdateRollback. These resources were not rolled back and may have drifted from the template. Run drift detection (DetectStackDrift) to review their current state before your next update."--resources-to-skipleft the stack without a reason.Unchanged here:
create-change-setwithout--change-set-typeon an existing stack answersAlreadyExistsExceptionin Floci, because Floci defaults the type toCREATE. AWS defaults it toUPDATE: on a missing stack, AWS answeredValidationError"Stack [name] does not exist".Checklist
./mvnw testpasses locally. The full suite was not run. I ran:./mvnw test -Dtest='io.github.hectorvent.floci.services.cloudformation.**': 3201 tests, 0 failures, 0 errors../mvnw -q checkstyle:check: passed.make partition-check: passed.make docs-check: passed.