Skip to content

feat(cloudformation): refuse updates after a failed rollback and add ContinueUpdateRollback - #4949

Merged
pgermosen merged 1 commit into
floci-io:mainfrom
mariotoffia:feat/cfn-continue-update-rollback
Oct 14, 2026
Merged

pgermosen merged 1 commit into
floci-io:mainfrom
mariotoffia:feat/cfn-continue-update-rollback

Conversation

@mariotoffia

@mariotoffia mariotoffia commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Closes #4945

When an update fails and the rollback of one of its resources fails too, the stack settles in UPDATE_ROLLBACK_FAILED. Floci now handles that state as AWS does.

Refused updates. UpdateStack and an UPDATE change set are refused with ValidationError Stack:<arn> is in UPDATE_ROLLBACK_FAILED state and can not be updated. Floci used to accept them and build the new update on resources whose state was unknown. DeleteStack stays allowed.

ContinueUpdateRollback. The new action takes StackName, ResourcesToSkip and ForceRollback:

  • It runs the rollback again for the resources still owed one. The stack first gets an UPDATE_ROLLBACK_IN_PROGRESS event with the reason User Initiated.
  • Each resource named in ResourcesToSkip is reported UPDATE_COMPLETE with the reason Resource skipped during UpdateRollback, and is not rolled back.
  • ForceRollback=true skips every UPDATE_FAILED resource of the stack the same way. When it skipped one, the completed stack's reason says the skipped resources may have drifted, with AWS's text.
  • Once every other resource has rolled back, the stack reaches UPDATE_ROLLBACK_COMPLETE. It gets back the template, parameters and outputs from before the failed update, and takes updates again.
  • If a restore fails again, the stack returns to UPDATE_ROLLBACK_FAILED, and a later call can retry it or skip it. An unexpected error during the retry settles the stack there too, so it is never left in UPDATE_ROLLBACK_IN_PROGRESS.

To make that possible, a failed rollback now keeps on the stack:

  • the state from before the update, as a StackUpdateSnapshot, which used to be a private record
  • the logical ids still owed a rollback

Both persist, so a retry works after a restart too. A retry reads them in UPDATE_ROLLBACK_IN_PROGRESS, and they are cleared once a rollback completes or the stack is deleted.

As on AWS:

  • A malformed call is refused with ValidationError before the stack is looked up. StackName must be present and match [a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]*, and every skip must match [a-zA-Z0-9]+|[a-zA-Z][-a-zA-Z0-9]*\.[a-zA-Z0-9]+. Every violation is reported in one message, StackName first.
  • ForceRollback=true with skips is refused with ValidationError ForceRollback and ResourcesToSkip are mutually exclusive, and a missing stack with Stack: <StackName> does not exist.
  • The call is refused from any other status, including UPDATE_ROLLBACK_IN_PROGRESS, with ValidationError RollbackUpdatedStack cannot be called from current stack status.
  • ResourcesToSkip is checked after the call is accepted. A bad skip returns the stack to UPDATE_ROLLBACK_FAILED with AWS's reason and touches no resource.

DescribeStackResource now returns ResourceStatusReason. That is where the skip reason shows.

What is covered

Legend: ✅ full, 🟡 partial, ❌ not implemented

Path Item Status Note
UpdateStack Stack in UPDATE_ROLLBACK_FAILED ✅ ValidationError with AWS's text
CreateChangeSet UPDATE change set on a stack in UPDATE_ROLLBACK_FAILED ✅ Same ValidationError
DeleteStack Stack in UPDATE_ROLLBACK_FAILED ✅ Unchanged: allowed
ContinueUpdateRollback Retry without skips ✅ Re-runs each owed restore. The stack ends UPDATE_ROLLBACK_COMPLETE, or UPDATE_ROLLBACK_FAILED again
ContinueUpdateRollback ResourcesToSkip of UPDATE_FAILED resources ✅ UPDATE_COMPLETE "Resource skipped during UpdateRollback"; the skipped resource keeps the state the failed update left it in
ContinueUpdateRollback ForceRollback ✅ Every UPDATE_FAILED resource is skipped; the stack reason names the possible drift. Refused together with ResourcesToSkip
ContinueUpdateRollback Malformed StackName or ResourcesToSkip ✅ ValidationError before the stack lookup, every violation in one message
ContinueUpdateRollback Stack that does not exist ✅ ValidationError Stack: <StackName> does not exist
ContinueUpdateRollback Stack in another status ✅ ValidationError "RollbackUpdatedStack cannot be called from current stack status"
ContinueUpdateRollback A skip not in the stack, or not UPDATE_FAILED ✅ 200, then UPDATE_ROLLBACK_FAILED with AWS's reason: "Resources with logicalIds [X] do not belong to stack S" or "Only the resources in UPDATE_FAILED state are allowed to be skipped"
ContinueUpdateRollback A skip naming a missing nested stack ✅ UPDATE_ROLLBACK_FAILED "Stack [X] does not exist"
ContinueUpdateRollback A skip of a nested stack's resource (Nested.Child) 🟡 Refused with a reason saying Floci does not support it; AWS would skip it
ContinueUpdateRollback Retrying the delete of a replacement whose rollback could not delete it ✅ The retry deletes it again; the stack ends UPDATE_ROLLBACK_COMPLETE once it is gone
ContinueUpdateRollback Retrying a restore that cannot be redone 🟡 A restore that spends its saved state on the first attempt fails again, so the resource must be skipped. Known cases: SQS, Pipes and the EC2 launch template. The CloudWatch dashboard, the case tested, keeps its state until a restore succeeds
ContinueUpdateRollback RoleARN, ClientRequestToken 🟡 Accepted, not validated, no effect, as on Floci's other stack operations
ContinueUpdateRollback UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS ❌ Not emitted, as for the first rollback
ContinueUpdateRollback Stack that failed under an earlier Floci version 🟡 Retries every UPDATE_FAILED resource and keeps the template it holds
ContinueUpdateRollback A DeleteStack racing the retry ❌ Not guarded, as for other operations
ContinueUpdateRollback A restart during the retry ❌ Not resumed, as for other operations
ContinueUpdateRollback A storage failure while submitting the retry ❌ The stack is not restored. Same as for other operations, which restore only when the executor is full
Rollback Status of a restored resource afterwards 🟡 Unchanged: CREATE_COMPLETE from the walker; AWS reports UPDATE_COMPLETE

Files

  • CloudFormationService:
    • continueUpdateRollback: validates the request, then checks the status and moves to UPDATE_ROLLBACK_IN_PROGRESS in one step under the stack's lock, so a racing update or second retry is refused. The rest runs on the operation executor.
    • validateContinueUpdateRollback: the StackName and ResourcesToSkip constraints, in AWS's message format.
    • resumeUpdateRollback: checks the skips (all UPDATE_FAILED resources when forced), reports the skipped resources, and re-runs the rollback walker over the owed resources. It records what is still owed before the walker and again after it, so a later failure never rolls back the same resource twice. An unexpected error settles the stack in UPDATE_ROLLBACK_FAILED.
    • invalidSkipReason: why AWS would fail the rollback over the skips.
    • finishUpdateRollback: shared by the first rollback and each retry. It keeps or clears the snapshot and the owed ids.
    • The update refusal now includes UPDATE_ROLLBACK_FAILED.
  • model/StackUpdateSnapshot: the record moved out of the service, registered for reflection.
  • model/Stack: updateRollbackSnapshot and updateRollbackResourceIds.
  • CloudFormationQueryHandler: the ContinueUpdateRollback action, including ForceRollback, with an empty <ContinueUpdateRollbackResult/>, and ResourceStatusReason in DescribeStackResource.
  • AwsQueryController: ContinueUpdateRollback in the CloudFormation action list.
  • provisioners/ReplacementCleanup: a rollback whose delete of the replacement fails now names it on the record, so a retry deletes it again instead of reporting that the type cannot roll back. A later provision drops the name; the replacement stays owed to the next committed cleanup or the stack delete, as before.
  • docs/services/cloudformation.md: the action row and a "Failed update rollbacks" section.
  • docs/services/index.md: the CloudFormation operation count, 35 to 36.

Tests:

  • CloudFormationContinueUpdateRollbackIntegrationTest (new, 14 tests):
    • updates and change sets refused while DeleteStack works
    • a retry without skips fails again
    • skipping the failed log group completes the rollback and unblocks updates
    • ForceRollback skips the failed resources, sets the drift reason and unblocks updates
    • a call on a stack in another status is refused
    • a bad skip fails the rollback again without touching a resource
    • a restore that threw is run again with the pre-update template
    • validation: a missing StackName, an empty one (length and pattern), one outside the pattern, a bad skip rejecting the whole list, and a missing StackName with a bad skip reported together
    • ForceRollback with skips refused without touching the stack
    • a missing stack named as given
  • CloudFormationServiceRollbackTest, 8 new tests:
    • the refusal
    • the persisted snapshot and owed ids
    • a stack without a snapshot
    • a nested skip
    • a restore that threw
    • an unexpected error during the retry settles the stack in UPDATE_ROLLBACK_FAILED, and a later call can continue it
    • an unexpected error after a skip no longer owes the skipped resource a rollback
    • a forced retry skips every UPDATE_FAILED resource and sets the drift reason
  • ReplacementCleanupTest, 3 new tests: a retry deletes the replacement the failed rollback left, a retry that fails again keeps it owed, and a later provision stops the retry while keeping it owed.
  • Tests that updated a stack in UPDATE_ROLLBACK_FAILED now continue the rollback first, since that update is refused:
    • CloudFormationPipesCleanupIntegrationTest: continues the rollback, skipping the pipe.
    • CognitoResourceServerCfnIntegrationTest (from feat(cloudformation): provision Cognito resource servers #4832), 6 tests: a plain retry where the restore can be redone, a skip of Server where the test needs the failed state kept, and one test now asserting the refusal.
  • EcsCfnProvisionerTest and ElbV2CfnProvisionerTest, one test each: a second rollback asserted it deleted nothing; it now retries the failed delete.

Type of change

  • Bug fix (fix:)
  • New feature (feat:)
  • Breaking change (feat!: or fix!:)
  • Docs / chore

AWS Compatibility

Measured with AWS CLI v2 2.37.7 in eu-west-1 on 2026-10-02. A stack held a custom resource whose rollback failed until told otherwise, and a bucket the update added with an invalid name.

  • update-stack and create-change-set --change-set-type UPDATE on the UPDATE_ROLLBACK_FAILED stack failed with ValidationError Stack:<arn> is in UPDATE_ROLLBACK_FAILED state and can not be updated. list-change-sets stayed empty.
  • continue-update-rollback on a stack in UPDATE_COMPLETE, CREATE_COMPLETE, UPDATE_ROLLBACK_COMPLETE or UPDATE_ROLLBACK_IN_PROGRESS failed with ValidationError "RollbackUpdatedStack cannot be called from current stack status".
  • An accepted call answered an empty <ContinueUpdateRollbackResult/>, then an UPDATE_ROLLBACK_IN_PROGRESS stack event "User Initiated".
  • A call without skips retried the resource's restore. It failed again, and the stack returned to UPDATE_ROLLBACK_FAILED.
  • --resources-to-skip with the resource gave UPDATE_COMPLETE "Resource skipped during UpdateRollback" without calling the handler. The stack then ended UPDATE_ROLLBACK_COMPLETE, and describe-stack-resource showed that status and reason.
  • Skips that were not UPDATE_FAILED, not in the stack, or of a missing nested stack were accepted with 200. The stack then returned to UPDATE_ROLLBACK_FAILED with the reasons listed in the table above.
  • After the skip, update-stack was accepted and updated from the template from before the failed update.

AWS also passed through UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS to delete the added bucket. Floci deletes it during the rollback, as it already did.

The same scenario ran with AWS CLI v2 2.37.7 against this branch. It used a log group whose rollback fails, because Floci has no failing custom resource handler.

Step Floci
wait stack-update-complete after the failing update Exits with "matched expected path: UPDATE_ROLLBACK_FAILED"; the stack reason is "The following resource(s) failed to roll back: [LogGroup]."
update-stack ValidationError with AWS's text
create-change-set --change-set-type UPDATE ValidationError with AWS's text
continue-update-rollback rc 0, then UPDATE_ROLLBACK_FAILED again
continue-update-rollback --resources-to-skip LogGroup rc 0. wait stack-rollback-complete rc 0. UPDATE_ROLLBACK_COMPLETE, and the resource shows UPDATE_COMPLETE "Resource skipped during UpdateRollback"
continue-update-rollback on UPDATE_ROLLBACK_COMPLETE ValidationError "RollbackUpdatedStack cannot be called from current stack status"
update-stack, then wait stack-update-complete rc 0

Measured again with AWS CLI v2 2.37.13 in eu-west-1 on 2026-10-13, for the request checks and ForceRollback:

  • A missing, empty or malformed StackName, and a skip outside its pattern, failed with HTTP 400 ValidationError before the stack lookup. All violations came in one message, StackName first. A 300-character skip was accepted: a skip has no length limit.
  • --force-rollback with --resources-to-skip failed with ValidationError "ForceRollback and ResourcesToSkip are mutually exclusive" and left the stack unchanged.
  • continue-update-rollback on a stack that does not exist failed with ValidationError "Stack: does not exist".
  • --force-rollback reported the failed resource UPDATE_COMPLETE "Resource skipped during UpdateRollback" without calling its handler. The stack ended UPDATE_ROLLBACK_COMPLETE with the pre-update template, and its reason, also on the completion event, was "One or more resources were skipped during UpdateRollback. These resources were not rolled back and may have drifted from the template. Run drift detection (DetectStackDrift) to review their current state before your next update."
  • A rollback completed through --resources-to-skip left the stack without a reason.

Unchanged here: create-change-set without --change-set-type on an existing stack answers AlreadyExistsException in Floci, because Floci defaults the type to CREATE. AWS defaults it to UPDATE: on a missing stack, AWS answered ValidationError "Stack [name] does not exist".

Checklist

  • ./mvnw test passes locally. The full suite was not run. I ran:
    • ./mvnw test -Dtest='io.github.hectorvent.floci.services.cloudformation.**': 3201 tests, 0 failures, 0 errors.
    • ./mvnw -q checkstyle:check: passed.
    • make partition-check: passed.
    • make docs-check: passed.
  • New or updated integration test added
  • Commit messages follow Conventional Commits

mariotoffia pushed a commit to mariotoffia/floci that referenced this pull request Oct 2, 2026
…a failed rollback and add ContinueUpdateRollback
@hectorvent hectorvent added cloudformation AWS CloudFormation enhancement New feature or request over-pr-limit Author has more than 2 open pull requests (see CONTRIBUTING.md) labels Oct 2, 2026
@github-actions github-actions Bot removed the over-pr-limit Author has more than 2 open pull requests (see CONTRIBUTING.md) label Oct 3, 2026
mariotoffia pushed a commit to mariotoffia/floci that referenced this pull request Oct 4, 2026
…a failed rollback and add ContinueUpdateRollback
mariotoffia pushed a commit to mariotoffia/floci that referenced this pull request Oct 4, 2026
…a failed rollback and add ContinueUpdateRollback
mariotoffia pushed a commit to mariotoffia/floci that referenced this pull request Oct 4, 2026
…a failed rollback and add ContinueUpdateRollback
mariotoffia pushed a commit to mariotoffia/floci that referenced this pull request Oct 4, 2026
…a failed rollback and add ContinueUpdateRollback
@mariotoffia
mariotoffia force-pushed the feat/cfn-continue-update-rollback branch from cd18ec0 to 3d24fbc Compare October 4, 2026 15:30
mariotoffia pushed a commit to mariotoffia/floci that referenced this pull request Oct 4, 2026
…a failed rollback and add ContinueUpdateRollback
@mariotoffia
mariotoffia force-pushed the feat/cfn-continue-update-rollback branch from 3d24fbc to 3109d12 Compare October 7, 2026 14:48
mariotoffia pushed a commit to mariotoffia/floci that referenced this pull request Oct 7, 2026
…a failed rollback and add ContinueUpdateRollback
@mariotoffia
mariotoffia force-pushed the feat/cfn-continue-update-rollback branch from 3109d12 to 50e1bc6 Compare October 8, 2026 05:29
mariotoffia pushed a commit to mariotoffia/floci that referenced this pull request Oct 8, 2026
…a failed rollback and add ContinueUpdateRollback
@mariotoffia
mariotoffia force-pushed the feat/cfn-continue-update-rollback branch 2 times, most recently from 3e14ed6 to 3ef0ae8 Compare October 13, 2026 04:05
@mariotoffia
mariotoffia marked this pull request as ready for review October 13, 2026 04:55
Copilot AI balanced review requested due to automatic review settings October 13, 2026 04:55
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Handle failed CloudFormation update rollbacks and add rollback retries

✨ Enhancement 🐞 Bug fix 🧪 Tests 📝 Documentation 🕐 40+ Minutes

Grey Divider

AI Description

• Refuse stack updates and UPDATE change sets until a failed rollback is resolved.
Diagram

graph TD
  A["Rollback failed"] --> B{"Action allowed?"} -->|Continue| D{"Valid skips?"} -->|Yes| E["Retry restores"] -->|Succeeded| F["Rollback complete"]
  B -->|Update| C["Reject update"]
  D -->|No| G["Rollback failed again"]
  E -->|Failed| G
  G --> A
Loading
High-Level Assessment

Persisting the pre-update snapshot and outstanding resource IDs is appropriate: rollback retries must work after the original operation finishes or the process restarts. Reconstructing that state from events would be less reliable and substantially more complex.

Files changed (15) +1028 / -63

Enhancement (5) +225 / -14
AwsQueryController.javaRoute ContinueUpdateRollback as a CloudFormation action +1/-0

Route ContinueUpdateRollback as a CloudFormation action

• Registers the new action in the shared Query controller’s CloudFormation action set.

src/main/java/io/github/hectorvent/floci/core/common/AwsQueryController.java

CloudFormationQueryHandler.javaExpose rollback continuation through the Query protocol +23/-0

Expose rollback continuation through the Query protocol

• Parses ResourcesToSkip, invokes the service, and returns a ContinueUpdateRollback XML response. DescribeStackResource now includes ResourceStatusReason so callers can see why a resource was skipped.

src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationQueryHandler.java

CloudFormationService.javaBlock unsafe updates and orchestrate rollback retries +166/-14

Block unsafe updates and orchestrate rollback retries

• Rejects updates in UPDATE_ROLLBACK_FAILED and accepts continuation only from that state. Validates skips asynchronously, retries outstanding restores, and retains or clears rollback state according to the outcome.

src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java

Stack.javaStore outstanding rollback state on stacks +11/-0

Store outstanding rollback state on stacks

• Adds a pre-update snapshot and logical IDs still owed a rollback to the persisted stack model.

src/main/java/io/github/hectorvent/floci/services/cloudformation/model/Stack.java

StackUpdateSnapshot.javaMake pre-update snapshots persistable +24/-0

Make pre-update snapshots persistable

• Moves the service’s private snapshot record into a serializable model that can survive restart and support later rollback retries.

src/main/java/io/github/hectorvent/floci/services/cloudformation/model/StackUpdateSnapshot.java

Bug fix (1) +35 / -6
ReplacementCleanup.javaRetry failed replacement deletions +35/-6

Retry failed replacement deletions

• Records a replacement whose rollback delete failed and attempts that delete again on rollback continuation. A subsequent new update clears the retry marker while retaining cleanup responsibility.

src/main/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ReplacementCleanup.java

Tests (7) +730 / -42
CloudFormationContinueUpdateRollbackIntegrationTest.javaExercise failed rollback recovery through Query requests +450/-0

Exercise failed rollback recovery through Query requests

• Tests update refusal, deletion, retry failure and success, skips, invalid skips, status reasons, template restoration, and a resource restore that succeeds on retry.

src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationContinueUpdateRollbackIntegrationTest.java

CloudFormationPipesCleanupIntegrationTest.javaRecover a failed Pipes rollback before updating +20/-4

Recover a failed Pipes rollback before updating

• Skips the failed pipe resource through ContinueUpdateRollback before testing that a later update preserves the prior pipe and handles orphan cleanup.

src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationPipesCleanupIntegrationTest.java

CloudFormationServiceRollbackTest.javaVerify persisted and legacy rollback retries +171/-3

Verify persisted and legacy rollback retries

• Tests update refusal, snapshot serialization and restoration, fallback for stacks without a stored snapshot, unsupported nested skips, and repeated provisioner restoration.

src/test/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationServiceRollbackTest.java

CognitoResourceServerCfnIntegrationTest.javaUse rollback continuation in Cognito recovery tests +28/-27

Use rollback continuation in Cognito recovery tests

• Replaces updates against failed-rollback stacks with retries or skips, and verifies that an update attempted before recovery is refused.

src/test/java/io/github/hectorvent/floci/services/cloudformation/CognitoResourceServerCfnIntegrationTest.java

EcsCfnProvisionerTest.javaExpect ECS replacement deletion to retry +2/-2

Expect ECS replacement deletion to retry

• Changes the repeated-rollback expectation to verify that an outstanding replacement delete fails again rather than being silently bypassed.

src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/EcsCfnProvisionerTest.java

ElbV2CfnProvisionerTest.javaVerify ELB replacement cleanup on retry +3/-6

Verify ELB replacement cleanup on retry

• Asserts that a second rollback deletes the outstanding replacement load balancer without deleting the restored prior one.

src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ElbV2CfnProvisionerTest.java

ReplacementCleanupTest.javaCover replacement deletion retry bookkeeping +56/-0

Cover replacement deletion retry bookkeeping

• Tests successful and repeated failed deletion retries, plus the behavior when a new update supersedes the failed rollback.

src/test/java/io/github/hectorvent/floci/services/cloudformation/provisioners/ReplacementCleanupTest.java

Documentation (2) +38 / -1
cloudformation.mdDocument failed rollback recovery +37/-0

Document failed rollback recovery

• Adds ContinueUpdateRollback to the operation list and explains update refusal, retries, resource skipping, compatibility with older stacks, and the nested-stack limitation.

docs/services/cloudformation.md

index.mdIncrease the CloudFormation operation count +1/-1

Increase the CloudFormation operation count

• Changes the documented Query operation count from 35 to 36.

docs/services/index.md

@qodo-code-review

qodo-code-review Bot commented Oct 13, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Action required

1. Deletion can race with rollback recovery ✗ Dismissed
Description
continueUpdateRollback submits a worker that retains the stack object, but neither that worker nor
DeleteStack excludes the other operation. If deletion removes the stack while the rollback worker
is still running, finishUpdateRollback can persist that same object again, leaving a deleted stack
in storage that reappears after restart.
Code

src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[R1103-1104]

+            return submitOperation(() -> RequestScopes.runAs(accountId, region,
+                    () -> resumeUpdateRollback(stack, skips, region)));
Relevance

●●● Strong

Team accepts lifecycle races that can orphan or resurrect resources, especially when asynchronous
cleanup persists stale state.

PR-#3761
PR-#4470

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The new worker receives the mutable stack after the compute lock is released. Deletion can
independently operate on that object, remove its live-map entry and delete its stored entry;
rollback completion still calls persistStack.

src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[1087-1104]
src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[1048-1061]
src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[2760-2773]
src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[2320-2332]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A rollback-recovery worker can continue using and persisting a stack after concurrent deletion removes it.
## Fix Focus Areas
- src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[1082-1114]
- src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[1029-1069]
- src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[2746-2774]
## Recommended Fix
Coordinate deletion and rollback recovery for the full operation, not just the initial status change. Prevent a rollback worker from mutating or persisting a stack once deletion has claimed it, and add a test that overlaps the two operations.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. A failed retry can strand the stack mid-rollback ✗ Dismissed
Description
resumeUpdateRollback runs on the operation executor, wrapped only in RequestScopes.runAs, with
no try/catch, and continueUpdateRollback discards the returned Future. Any unchecked exception
from persistStack (storage I/O), finishUpdateRollback or the skip loop is never logged and
leaves the stack in UPDATE_ROLLBACK_IN_PROGRESS. ContinueUpdateRollback refuses that status, and
refusesUpdate refuses every _IN_PROGRESS status, so a further retry or update cannot recover the
stack.
Code

src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[R2188-2190]

+    void resumeUpdateRollback(Stack stack, List<String> resourcesToSkip, String region) {
+        String invalidSkip = invalidSkipReason(stack, resourcesToSkip);
+        if (invalidSkip != null) {
Relevance

●●● Strong

Unhandled rollback exceptions leaving terminal state inconsistent match accepted precedents
requiring failure propagation and explicit rollback-failure tracking.

PR-#4728
PR-#4731

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The submitted lambda calls resumeUpdateRollback directly and nothing reads the Future. In
contrast, executeTemplate wraps its body in a catch that always moves the stack to a terminal
status. The new entry point refuses every status except UPDATE_ROLLBACK_FAILED.

src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[1101-1105]
src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[1092-1094]
src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[1916-1941]
src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[1149-1155]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`resumeUpdateRollback` runs on the executor with no exception handling, and nobody reads its Future. An unchecked exception leaves the stack in UPDATE_ROLLBACK_IN_PROGRESS for good and is never logged. ContinueUpdateRollback refuses that status, so the stack cannot recover.

## Fix Focus Areas
- src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[2188-2227]
- src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[1101-1105]

## Recommended Fix
Wrap the body of `resumeUpdateRollback` in a try/catch on `Exception`. In the catch:
- Log the exception with its stack trace.
- Set the stack's status reason to the exception message, add an `UPDATE_ROLLBACK_FAILED` stack event, and set the status to `UPDATE_ROLLBACK_FAILED`.
- Keep the existing `updateRollbackSnapshot` and `updateRollbackResourceIds`, or the ones derived for a legacy stack, so a later ContinueUpdateRollback can retry.
- Call `persistStack` as a best effort, inside its own try/catch.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. A restart can strand rollback recovery ✗ Dismissed
Description
continueUpdateRollback persists UPDATE_ROLLBACK_IN_PROGRESS before submitting its worker, while
startup only reloads the stored status and does not resume the operation. If the process stops after
that write but before the worker completes, the stack remains in progress after restart and another
recovery call is refused because it requires UPDATE_ROLLBACK_FAILED.
Code

src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[R1102-1104]

+            persistStack(stack);
+            return submitOperation(() -> RequestScopes.runAs(accountId, region,
+                    () -> resumeUpdateRollback(stack, skips, region)));
Relevance

●●● Strong

Accepted patterns prioritize persisted rollback state and recovery across failures; restart-stranded
operations are a concrete reliability gap.

PR-#4728
PR-#4731

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The accepted request stores the in-progress status before scheduling asynchronous work. Startup
reloads stored stacks without scheduling recovery, and the new status check rejects a subsequent
ContinueUpdateRollback call.

src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[1092-1104]
src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[137-145]
src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[1087-1099]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A restart during ContinueUpdateRollback can leave a persisted in-progress stack with no worker to finish it and no accepted retry path.
## Fix Focus Areas
- src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[137-145]
- src/main/java/io/github/hectorvent/floci/services/cloudformation/CloudFormationService.java[1082-1104]
## Recommended Fix
On startup, detect an interrupted ContinueUpdateRollback operation and either safely resume it from the stored snapshot or return it to a retryable failed state. Persist enough operation identity to distinguish this case from other in-progress rollbacks, and test a restart between request acceptance and worker completion.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
✅ Compliance rules (platform): 8 rules
✅ Cross-repo context — repo relationships
  Explored: repo: floci-io/floci-ui (sha: 70d1b2d6) — View relationship
Review mode: Auto: 🧠 Deep: Stateful API change spans rollback, persistence, concurrency, and cleanup paths.

Grey Divider

Tip of the day
💡 Did you know, you can switch off images and animations for a plain-text comment

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new Query action does not synchronously validate required and constrained AWS request parameters.

1 open finding
What changed in this PR

Adds AWS-compatible recovery for CloudFormation stacks in UPDATE_ROLLBACK_FAILED.

Changes:

  • Implements ContinueUpdateRollback, resource skipping, and persisted rollback snapshots.
  • Refuses updates and update change sets while rollback recovery is required.
  • Adds protocol responses, documentation, and comprehensive rollback tests.
File Description
ReplacementCleanupTest.java Tests replacement-delete retries.
ElbV2CfnProvisionerTest.java Updates ELB retry expectations.
EcsCfnProvisionerTest.java Updates ECS retry expectations.
CognitoResourceServerCfnIntegrationTest.java Uses rollback recovery before later updates.
CloudFormationServiceRollbackTest.java Tests snapshots, skips, persistence, and retries.
CloudFormationPipesCleanupIntegrationTest.java Adds recovery before pipe updates.
CloudFormationContinueUpdateRollbackIntegrationTest.java Covers the new action end to end.
ReplacementCleanup.java Tracks and retries undeleted replacements.
StackUpdateSnapshot.java Defines persisted pre-update state.
Stack.java Stores pending rollback metadata.
CloudFormationService.java Implements refusal and rollback continuation logic.
CloudFormationQueryHandler.java Adds the Query action and status reason response.
AwsQueryController.java Registers the new CloudFormation action.
docs/​services/​index.md Updates the operation count.
docs/​services/​cloudformation.md Documents failed rollback recovery.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@mariotoffia
mariotoffia force-pushed the feat/cfn-continue-update-rollback branch from 3ef0ae8 to 73214f5 Compare October 13, 2026 05:45
@mariotoffia
mariotoffia requested a balanced review from Copilot October 13, 2026 05:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unexpected failures can retain stale rollback work and make a later recovery retry fail or become unrecoverable.

1 open finding
1 resolved since last review

🧠 Review effort: Balanced

@mariotoffia
mariotoffia force-pushed the feat/cfn-continue-update-rollback branch from 73214f5 to a148188 Compare October 13, 2026 06:03
@mariotoffia
mariotoffia requested a balanced review from Copilot October 13, 2026 06:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Stateful asynchronous rollback and persistence semantics warrant final human review despite extensive coverage.

0 open findings

1 resolved since last review

🧠 Review effort: Balanced

@pgermosen pgermosen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mariotoffia really thorough work, and the AWS measurements made it easy to check.

The Cognito resource server paragraph in the docs still describes the old flow, though. It points a failed restoration at a later update, and says an update can stay stuck in cleanup. Now that updates are refused after a failed rollback, that retry is ContinueUpdateRollback. A skipped server comes out UPDATE_COMPLETE, so the next update just drops its pending restoration and the stuck-cleanup case can't happen. You already turned its test into the refusal test, so that sentence should go too.

In finishUpdateRollback the failed branch still sets the status before the reason and the event. Please flip it to match the order you used everywhere else.

@pgermosen pgermosen added the waiting-author Review posted; waiting on the PR author to respond label Oct 13, 2026
@mariotoffia
mariotoffia force-pushed the feat/cfn-continue-update-rollback branch from a148188 to 66340cd Compare October 14, 2026 03:52
@mariotoffia

Copy link
Copy Markdown
Contributor Author

@pgermosen both addressed in 66340cd (rebased on current main):

  • docs/services/cloudformation.md, Cognito resource servers: the retry for a failed in-place restoration is now ContinueUpdateRollback, and the UPDATE_COMPLETE_CLEANUP_IN_PROGRESS sentence is gone. A skipped server ends UPDATE_COMPLETE, so the next committed update drops its pending restoration, and anUpdateThatLeavesAPendingRollbackAloneIsRefused covers the refusal.
  • finishUpdateRollback: the failed branch now records the snapshot and owed ids, then the reason and the stack event, and sets UPDATE_ROLLBACK_FAILED last, the same order as the success branch and resumeUpdateRollback.

CloudFormation package: 3201 tests, 0 failures. Checkstyle, docs-check and partition-check pass.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Retry bookkeeping and pre-worker storage failures can leave otherwise recoverable stacks stuck.

2 open findings

🧠 Review effort: Balanced

@mariotoffia
mariotoffia force-pushed the feat/cfn-continue-update-rollback branch from 66340cd to 21d4986 Compare October 14, 2026 08:31
@mariotoffia
mariotoffia requested a balanced review from Copilot October 14, 2026 08:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The persistent asynchronous rollback state machine warrants final human review despite strong test coverage.

1 open finding
2 resolved since last review

🧠 Review effort: Balanced

@mariotoffia
mariotoffia force-pushed the feat/cfn-continue-update-rollback branch from 21d4986 to b011b00 Compare October 14, 2026 08:43
@mariotoffia
mariotoffia requested a balanced review from Copilot October 14, 2026 08:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes asynchronous, persisted rollback state across many CloudFormation resource lifecycles and warrants final human review.

0 open findings

1 resolved since last review

🧠 Review effort: Balanced

@pgermosen pgermosen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mariotoffia thanks, both are in: the resource-server docs now point a failed restoration at ContinueUpdateRollback without the stuck-cleanup sentence, and the failed branch of finishUpdateRollback sets the status last. Recording only the walker's failures before finishing is a good catch too.

@pgermosen pgermosen removed the waiting-author Review posted; waiting on the PR author to respond label Oct 14, 2026
@pgermosen
pgermosen merged commit c4a809f into floci-io:main Oct 14, 2026
28 of 30 checks passed
pgermosen added a commit to jwoa95/floci that referenced this pull request Oct 14, 2026
…nueUpdateRollback in schedules

floci-io#4949 refuses an update over UPDATE_ROLLBACK_FAILED until ContinueUpdateRollback finishes the
rollback, as AWS does. floci-io#4834, merged minutes earlier, had seven tests that update over that
status, and main has failed them since c4a809f.

Six are in CloudFormationSchedulerScheduleIntegrationTest. They now reach their next update the
way AWS requires: ContinueUpdateRollback, which retries the schedule's own rollback, or, where the
replacement's delete has to keep failing, a skip of the schedule. The two whose update was meant to
be blocked by the pending rollback now check that the update is refused.

The seventh shows a real clash. floci-io#4949 made a second ReplacementCleanup.rollback retry the delete
the first one could not make, which is ContinueUpdateRollback's job. The Scheduler provisioner also
calls rollbackUpdate inside an update, to undo its own failed call, so after a skip a failed group
move deleted the earlier orphan there and then, and a failure of that delete would have failed the
move's restore. A schedule update now ends that retry first, through
ReplacementCleanup.endRollbackRetry, so the orphan stays owed for the next committed update or the
stack delete, as both changes document.
pgermosen added a commit to jwoa95/floci that referenced this pull request Oct 14, 2026
…nueUpdateRollback

floci-io#4949 refuses an update over UPDATE_ROLLBACK_FAILED until ContinueUpdateRollback finishes the
rollback, as AWS does. floci-io#4834, merged minutes earlier, had seven tests that update over that
status, and main has failed them since c4a809f.

Six are in CloudFormationSchedulerScheduleIntegrationTest. They now reach their next update the
way AWS requires: ContinueUpdateRollback, which retries the schedule's own rollback, or, where the
replacement's delete has to keep failing, a skip of the schedule. The two whose update was meant to
be blocked by the pending rollback now check that the update is refused.

The seventh shows a real clash. floci-io#4949 made a second ReplacementCleanup.rollback retry the delete
the first one could not make, which is ContinueUpdateRollback's job. The Scheduler provisioner also
calls rollbackUpdate inside an update, to undo its own failed call, so after a skip a failed group
move deleted the earlier orphan there and then, and a failure of that delete would have failed the
move's restore. An update now ends that retry as it starts, through
ReplacementCleanup.endRollbackRetry, so the orphan stays owed for the next committed update or the
stack delete, as both changes document.

Three more provisioners roll an update back before ReplacementCleanup.record has cleared the retry,
and each returns as soon as the retry ran, skipping its own restore. CloudWatchDashboard and
IotAuthorizer unwind a failed in-place update that way, and CognitoResourceServer's failed attempt
is rolled back the same way, so after a skip the update's snapshot was never put back while the
resource was reported restored. They end the retry as an update starts too.
pgermosen added a commit that referenced this pull request Oct 14, 2026
* fix(ses): delete by id and filter by source in /_aws/ses

The inspection endpoint follows LocalStack's contract, as
docs/services/ses.md says, but differed from it in two ways. LocalStack
documents that "The query parameter id can be used to delete only a
specific message", and that "Query parameters id and email can be used
to filter by message ID and message source respectively".

- DELETE /_aws/ses?id=<id> cleared every message. It now deletes only
  that message, in whichever region it was sent. An unknown id deletes
  nothing.
- GET /_aws/ses?email=<address> matched the address against each
  message's ToAddresses. It now matches the message's Source, as
  LocalStack does.

GET also listed messages in storage hash order. It now lists them
oldest first by Timestamp, the order LocalStack keeps them in.

Floci's own tests read Cognito codes and other mail with
?email=<recipient>. They now list the mailbox and pick the recipient's
messages out of it, through a new SesMailbox test helper in src/test and
the same filtering in the Java and Node compatibility tests. A raw
send's mailbox entry shows RawData without its recipients, so the test
for recipients read from a raw message's headers checks the stored
record.

* fix(codeartifact): run the pypiserver sidecar under an init process so a stop ends it at once

The pypiserver image's entrypoint execs pypi-server, which installs no SIGTERM handler. The kernel
gives PID 1 no default signal actions, so the container ignored every stop request and went only
when Docker sent SIGKILL at the end of the 5-second timeout: each reset, repository delete and
shutdown waited that long for every pypi repository's container, one after another.

ContainerBuilder gains withInit(), carried on ContainerSpec as init and passed to Docker as
HostConfig.Init (docker run --init), and PypiserverSidecarManager uses it. Docker's init runs as
PID 1 and forwards the SIGTERM, so a stop now returns in a fraction of a second. Verdaccio already
handles the signal itself and is unchanged.

* fix(firehose): check buffers every second so delivery follows the buffering interval

AWS delivers a Firehose buffer once the first buffer condition is met. Floci checked for elapsed
buffering intervals on a 10-second tick, so a buffer was delivered up to 10 seconds after its
IntervalInSeconds ended: a 5-second interval landed anywhere from 5 to 15 seconds after the first
record. The default tick is now one second; the check is a scan of the in-memory buffers.

FirehoseFlushIntegrationTest's profile only set the same one-second tick, so it is removed and the
class shares the default test application. The compat tests that wait for a delivery checked S3
every 2 or 5 seconds; against Floci they now check every 250 ms, and keep their cadence against
real AWS.

* test: make subclass mocks the default so mocking stays cheap in a loaded test JVM

Mockito's default inline mock maker redefines every class it mocks in the running JVM. A
redefinition gets slow once the test JVM has loaded the Quarkus applications of the classes that
ran before, which in CI is most of the run: a unit class's first test spent seconds building its
mocks, and EmulatorLifecycleTest, which mocks some fifty types, spent 22 seconds. Subclass mocks
generate a subclass instead and redefine nothing. After twelve Quarkus applications in one JVM, the
fourteen Mockito classes whose first test was slowest in CI take 20.5 seconds instead of 41.9, and
their first tests 6.9 seconds instead of 27.8.

mockito-extensions/org.mockito.plugins.MockMaker makes the subclass mock maker the default. A
subclass cannot extend a final class, record or enum, so those mocks go through
InlineMocks.inlineMock and inlineSpy, or @mock(mockMaker = MockMakers.INLINE), and
EmulatorLifecycleTest's static mock of Quarkus asks for the inline maker. The ContainerSpec mocks
a mocked ContainerBuilder.Builder returned were opaque values, so they are real specs now.

* fix(cloudformation): leave the last rollback's failed delete to ContinueUpdateRollback

#4949 refuses an update over UPDATE_ROLLBACK_FAILED until ContinueUpdateRollback finishes the
rollback, as AWS does. #4834, merged minutes earlier, had seven tests that update over that
status, and main has failed them since c4a809f.

Six are in CloudFormationSchedulerScheduleIntegrationTest. They now reach their next update the
way AWS requires: ContinueUpdateRollback, which retries the schedule's own rollback, or, where the
replacement's delete has to keep failing, a skip of the schedule. The two whose update was meant to
be blocked by the pending rollback now check that the update is refused.

The seventh shows a real clash. #4949 made a second ReplacementCleanup.rollback retry the delete
the first one could not make, which is ContinueUpdateRollback's job. The Scheduler provisioner also
calls rollbackUpdate inside an update, to undo its own failed call, so after a skip a failed group
move deleted the earlier orphan there and then, and a failure of that delete would have failed the
move's restore. An update now ends that retry as it starts, through
ReplacementCleanup.endRollbackRetry, so the orphan stays owed for the next committed update or the
stack delete, as both changes document.

Three more provisioners roll an update back before ReplacementCleanup.record has cleared the retry,
and each returns as soon as the retry ran, skipping its own restore. CloudWatchDashboard and
IotAuthorizer unwind a failed in-place update that way, and CognitoResourceServer's failed attempt
is rolled back the same way, so after a skip the update's snapshot was never put back while the
resource was reported restored. They end the retry as an update starts too.

---------

Co-authored-by: Priamo <pgermosen@gmail.com>
@hectorvent

Copy link
Copy Markdown
Member

🎉 This PR is included in version 2.3.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cloudformation AWS CloudFormation enhancement New feature or request released

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEAT] CloudFormation: ContinueUpdateRollback, and refuse updates in UPDATE_ROLLBACK_FAILED

4 participants