Skip to content

[BUG] CloudFormation DynamoDB table: a retried rollback replays a committed update, and a non-object TimeToLiveSpecification disables TTL #5670

Closed

Description

@mariotoffia

Service

CloudFormation (AWS::DynamoDB::Table), DynamoDbCfnProvisioner.

AWS API Action

CreateStack, UpdateStack, ContinueUpdateRollback

Follow-up to #4951 (closes #4950). Copilot raised both findings in its second review of #4951 (snapshot, non-object block). #4951 merged as a5228a0 before the fix was pushed, so main still has both bugs.

Expected behavior

  1. A committed table update is never rolled back. After an update commits, a later rollback of the stack, including a retried one through ContinueUpdateRollback, leaves the table as that update left it.

  2. A TimeToLiveSpecification that is not an object fails model validation. Measured in eu-west-1 on 2026-10-14, a CreateStack fails pre-deployment with VALIDATION_ERROR and the stack ends ROLLBACK_COMPLETE with no table:

    • YAML TimeToLiveSpecification: false: "Property [TimeToLiveSpecification] expected type: JSONObject, found: String".
    • JSON "TimeToLiveSpecification": true: the same message, "found: String". AWS types every scalar as String here.
    • A list: "Property [TimeToLiveSpecification] expected type: JSONObject, found: JSONArray".

    {"Ref": "AWS::NoValue"} is not a value; it removes the property.

Actual behavior

  1. The snapshot of a committed update stays on the resource and is replayed.
    • An in-place update records the table's tags, stream and TTL in CfnRollback.DYNAMODB_TABLE_UPDATE_SNAPSHOT_ATTR, so a failed update can restore them.
    • Nothing removes that record when the update commits.
    • In one sequence the committed resource is put back with the stale record still on it: a later replacing update whose new table cannot be deleted.
      • The rollback walker (CloudFormationService.rollbackUpdatedResources, line 2412) reports UPDATE_FAILED once.
      • Then ContinueUpdateRollback calls rollbackUpdate on the committed resource.
      • That restores the stale record: the table's tags go back to their values before the committed update, and the stack reports UPDATE_ROLLBACK_COMPLETE.
  2. A non-object block is treated as removed.
    • provisionTable (line 126) reads the block with engine.resolveNode and sets ttlDeclared = ttlSpec != null && ttlSpec.isObject().
    • A boolean, a string or a list therefore counts as "not declared":
      • On create, the stack reaches CREATE_COMPLETE without TTL.
      • On an update from a template that declared the block, ctx.previouslyDeclared("TimeToLiveSpecification") makes the provisioner disable TTL, and the stack reaches UPDATE_COMPLETE.

Reproduction

Non-object block, against main:

Resources:
  Orders:
    Type: AWS::DynamoDB::Table
    Properties:
      AttributeDefinitions: [{AttributeName: pk, AttributeType: S}]
      KeySchema: [{AttributeName: pk, KeyType: HASH}]
      BillingMode: PAY_PER_REQUEST
      TimeToLiveSpecification: false
aws --endpoint-url http://localhost:4566 cloudformation create-stack --stack-name ttl-nonobj --template-body file://nonobject.yaml
aws --endpoint-url http://localhost:4566 cloudformation describe-stacks --stack-name ttl-nonobj --query 'Stacks[0].StackStatus'
# Floci: CREATE_COMPLETE. AWS: ROLLBACK_COMPLETE with the VALIDATION_ERROR above.

Deploying first with TimeToLiveSpecification: {AttributeName: expiresAt, Enabled: true} and then updating to false gives UPDATE_COMPLETE, and describe-time-to-live then answers DISABLED.

The stale snapshot needs a replacement table that cannot be deleted, so the reproduction is a fault-injected integration test (the second test under "Fix"). It runs:

  1. Create the table with tag env=dev.
  2. Update the tag to env=prod. This is in place and commits.
  3. Make a replacing update (a new TableName, with TTL). The new table is created, then its TTL call fails and the cleanup DeleteTable fails. The stack ends UPDATE_ROLLBACK_FAILED, which is correct.
  4. Run ContinueUpdateRollback. Today the stack ends UPDATE_ROLLBACK_COMPLETE and the table's tags are back to env=dev.

Environment

Fix

The fix is commit 00daf8d on mariotoffia:fix/cfn-dynamodb-table-ttl. It applies cleanly onto a5228a0 and touches only the three files below.

1. Drop the snapshot when the update commits

The engine calls completeUpdate on every resource after a commit (CloudFormationService, line 2121), and calls clearUpdate only when the result is applicable. CognitoCfnProvisioner.completeUpdate drops its snapshot the same way; the table has no replacement cleanup to run, so it returns notApplicable(). Add an override to DynamoDbCfnProvisioner, after retainsFailedUpdateState:

/**
 * The engine calls this for every resource once a stack update commits. A committed update is
 * never rolled back, so its snapshot is dropped here: left on the resource, a later failed update
 * would restore it and a retried rollback would put the table back to its state before this one.
 */
@Override
public UpdateCleanupResult completeUpdate(StackResource resource) {
    resource.getAttributes().remove(CfnRollback.DYNAMODB_TABLE_UPDATE_SNAPSHOT_ATTR);
    return UpdateCleanupResult.notApplicable();
}

Not part of this fix:

  • No clearUpdate override: the engine never calls it after a not-applicable result.
  • No ReplacementCleanup.recordOrphan: rolling back a replacing table update is still unsupported. After the fix, ContinueUpdateRollback in step 4 ends UPDATE_ROLLBACK_FAILED again with "Rollback is not implemented for AWS::DynamoDB::Table", with the committed tags kept. ContinueUpdateRollback --resources-to-skip Orders completes it. AWS instead retries the delete of the new table.

2. Reject a block that is not an object

In provisionTable, replace lines 126-127:

-        JsonNode ttlSpec = props != null ? engine.resolveNode(props.get("TimeToLiveSpecification")) : null;
-        boolean ttlDeclared = ttlSpec != null && ttlSpec.isObject();
+        JsonNode ttlSpec = props != null
+                ? engine.resolveNodeOmittingNoValue(props.get("TimeToLiveSpecification")) : null;
+        boolean ttlDeclared = ttlSpec != null && !ttlSpec.isMissingNode() && !ttlSpec.isNull();
+        if (ttlDeclared && !ttlSpec.isObject()) {
+            // As AWS reports it: scalars are typed String, so a boolean is "found: String" too.
+            throw new AwsException("InvalidRequest", "Model validation failed (#/TimeToLiveSpecification:"
+                    + " expected type: JSONObject, found: " + (ttlSpec.isArray() ? "JSONArray" : "String") + ")",
+                    400);
+        }

Why each piece is there:

  • resolveNodeOmittingNoValue is required. resolveNode leaves {"Ref": "AWS::NoValue"} as a non-object, which would now be rejected. resolveNodeOmittingNoValue returns a MissingNode for it, including inside Fn::If branches, so AWS::NoValue keeps meaning "removed".
  • The check sits before any DynamoDB call, so nothing on the table changes.
  • A JSON null is treated as omitted. This was not probed on AWS.

Tests

DynamoDbCfnProvisionerTest:

  • In the ctx() mock, stub engine.resolveNodeOmittingNoValue(any()) as identity, as the S3 and Scheduler provisioner tests do.
  • nonObjectTimeToLiveFailsModelValidationInsteadOfDisablingIt: a @ParameterizedTest with @CsvSource cases false/String, "invalid"/String, and a list/JSONArray.
    • It runs on an update context whose previous template declared TTL.
    • It asserts the message and verifyNoInteractions(dynamoDb).
  • committedUpdateLeavesNoSnapshotToRollBackTo: after an in-place tag update, completeUpdate is not applicable and a following rollbackUpdate returns false.

CloudFormationDynamoDbTimeToLiveIntegrationTest:

  • removingTheTimeToLiveBlockDisablesIt becomes @ParameterizedTest @ValueSource(strings = {"", TTL_NO_VALUE}), with TTL_NO_VALUE = {"Ref": "AWS::NoValue"}, so AWS::NoValue keeps disabling.
  • retryingAFailedReplacementRollbackKeepsTheCommittedTags: the four steps above.
    • It uses the existing failNextUpdateTimeToLive, and a one-shot deleteTable(anyString(), anyString()) failure on the DynamoDbService spy, with the real methods restored in finally.
    • Then it asserts:
      • UPDATE_ROLLBACK_FAILED after the continue.
      • Tags still env=prod.
      • UPDATE_ROLLBACK_COMPLETE after continueUpdateRollback(stack, "Orders").
    • Then it deletes the orphaned replacement table and the stack.
  • Helper continueUpdateRollback(String stack, String... resourcesToSkip).

Before the fix, five of these fail; after it, all pass. Gates run on 00daf8d:

  • ./mvnw -q checkstyle:check: passed.
  • ./mvnw test -Dtest='io.github.hectorvent.floci.services.cloudformation.**,io.github.hectorvent.floci.services.dynamodb.backend.**': 3485 tests, 0 failures, 12 skipped.
  • Against amazon/dynamodb-local:3.3.1: DynamoDbLocalBackendIntegrationTest 12 tests and CloudFormationDynamoDbTimeToLiveIntegrationTest 11 tests, 0 failures.
  • make partition-check and make docs-check: passed.

Activity

  1. AndreKurait commented on Nov 4, 2026

    @AndreKurait
    Contributor

    The invalid TTL block finding is fixed by merged #7286. The committed-snapshot finding remains on main at 27057e2: a fresh provisioner regression performs a successful tag update, calls completeUpdate, then confirms rollback still restores the earlier state.

    Existing #7448 implements the missing commit cleanup and includes the reported retried-stack-rollback sequence. Keep this issue open until that fix lands; no duplicate implementation is needed. This new reproduction is a local unit test, not a live AWS or current-PR runtime qualification.

  2. pgermosen commented on Nov 10, 2026

    @pgermosen
    Member

    Both findings are fixed on main: #7286 refuses a TimeToLiveSpecification that is not an object, and #7448 drops the update snapshot once the update commits, so a retried rollback no longer replays it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcloudformationAWS CloudFormationdynamodbAmazon DynamoDB

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions