Skip to content

test(rds): bound Docker requests without replaying creates - #8194

Merged
pgermosen merged 5 commits into
floci-io:mainfrom
AndreKurait:test/rds-docker-request-budget-main-20261112
Nov 13, 2026
Merged

pgermosen merged 5 commits into
floci-io:mainfrom
AndreKurait:test/rds-docker-request-budget-main-20261112

Conversation

@AndreKurait

@AndreKurait AndreKurait commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A slow synchronous PostgreSQL startup can exceed the SDK's read deadline. Replaying CreateDBInstance then reports DBInstanceAlreadyExists although the first request created the database. A prior loopback probe reproduced this in 30.9 seconds.

Use a separate Docker-fixture client with a 90-second total request budget and one attempt; the default shared client stays unchanged. Creation is covered by cleanup. A server-rejected create may leave no instance, but an ambiguous client failure waits for late publication and deletion within a deadline. Each delete request uses the remaining budget. Unconfirmed cleanup is retained as a suppressed error on the original create failure.

Original upgrade rejection, metadata, endpoint, SQL version and data-preservation assertions remain byte-identical. The engine-upgrade method has an explicit five-minute JUnit timeout, allowing its bounded create and cleanup waits to preserve the original create failure. No production code changes.

Type of change

  • Docs / chore: test fixture correction

AWS Compatibility

Test-only correction; no new AWS behavior or protocol claim.

Testing

The timeout-only follow-up c5441b082 compiled all production and test sources and passed the five cleanup regressions plus full Checkstyle. The real-Docker lifecycle was not rerun for this annotation-only change.

The previous head b3becaa738, qualified against main babf8419f4, passed 17 tests across four classes, with zero failures/errors/skips: five cleanup regressions, eleven existing SDK consumer tests and the complete real-Docker PostgreSQL engine-upgrade scenario. Java 25 packaging, full Checkstyle, docs, partition, CloudFormation property-type and whitespace gates passed. Both observed containers and the actual mounted volume were verified absent after teardown.

Two negative controls fail with the old immediate-not-found cleanup behavior: late publication is not retried, and unresolved cleanup is not reported. The corrected source was restored before final qualification. The published head's merge into that main produces the exact qualified tree.

Docker images were cached. Forced cold-start timing, real-Docker timeout fault injection, the full repository suite, native builds and live AWS were not run. The late-publication/error controls use an isolated mocked SDK client; the real-Docker run exercises the full successful lifecycle.

Risk and rollback

The request policy applies only to this Docker fixture. Failed or ambiguous creation can now spend a bounded cleanup interval rather than declaring absence immediately. Other deletion failures remain visible. Revert this test-only PR to restore the previous behavior.

Checklist

  • Full repository test suite: focused suites passed instead.
  • Existing integration scenario retained, cleanup regressions added.
  • Conventional Commits, signed commits and DCO.
  • At most two ready and four total open authored PRs.

Use a separate single-attempt client for synchronous Docker startup and cover creation with the fixture cleanup scope. Keep the default client and scenario assertions unchanged.

Signed-off-by: Andre Kurait <akurait@amazon.com>
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Bound Docker RDS requests without retrying instance creation

🧪 Tests 🐞 Bug fix 🕐 20-40 Minutes

Grey Divider

AI Description

• Give the Docker upgrade test a 90-second, single-attempt RDS client for slow PostgreSQL startup.
• Keep the shared client’s behavior unchanged and preserve the upgrade assertions.
• Attempt cleanup even when creation fails, ignoring only a missing test-owned instance.
Diagram

graph TD
  T["Docker upgrade test"] --> B["Shared client builder"] --> C["Single-attempt client"] --> R["Local RDS API"] --> P[("Docker PostgreSQL")]
  B --> D["Default RDS client"]
Loading
High-Level Assessment

The dedicated client is appropriate: it bounds slow Docker startup without changing other tests’ client behavior, while a single attempt avoids replaying instance creation. Broadening the shared client’s timeout or retry policy would unnecessarily affect other fixtures.

Files changed (2) +32 / -8

Bug fix (1) +14 / -6
RdsEngineVersionDockerIntegrationTest.javaUse bounded client and cover creation with cleanup +14/-6

Use bounded client and cover creation with cleanup

• Switches the Docker upgrade scenario to the dedicated RDS client and moves instance creation inside the existing cleanup scope. Deletion ignores only DbInstanceNotFoundException; upgrade, SQL, and data-preservation assertions remain unchanged.

src/test/java/io/floci/aws/services/rds/RdsEngineVersionDockerIntegrationTest.java

Other (1) +18 / -2
RdsRequests.javaAdd a single-attempt Docker RDS client +18/-2

Add a single-attempt Docker RDS client

• Extracts common RDS client configuration into a builder while leaving the default client unchanged. Adds a Docker-specific client with 90-second socket and total API-call timeouts and one SDK attempt.

src/test/java/io/floci/aws/testing/RdsRequests.java

@qodo-code-review

qodo-code-review Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Remediation recommended

1. Timed-out creates leave databases behind ✓ Resolved

Resolving commit: b3becaa

Resolution reason
Teardown now distinguishes server-rejected creates (RdsException, single delete tolerating
not-found) from ambiguous client failures (bounded retry of delete until success within the budget,
with cleanup failures preserved on the original error), so timed-out creates that publish late are
cleaned up.
Description
The DbInstanceNotFoundException catch treats a failed immediate delete as proof that
createDBInstance installed nothing, even though the server can still be starting the container. If
the 90-second client deadline expires before the server publishes the instance, teardown sees
not-found and stops; the server can subsequently publish the instance with its container and volume
still present.

Grey Divider

Context sources
❌ Tickets: failed to load — Ticket collection could not establish complete evidence
✅ Compliance rules (platform): 15 rules
✅ Cross-repo context — repo relationships
Review mode: Auto: ⚖️ Balanced: AWS timeout, retry, and cleanup changes warrant careful single-pass review.

Grey Divider

Tip of the day
💡 Did you know, you can start a comment with 'qodo' or '@qodo' to chat about any finding

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Wait for late publication before deleting after ambiguous client failure. Keep cleanup bounded and preserve failures on the original creation error. Cover late publication and unresolved cleanup with regression controls.

Signed-off-by: Andre Kurait <AndreKurait@gmail.com>

@Sekator778 Sekator778 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ran both tests on b3becaa and a create stalled for 200 s against a stub: one CreateDBInstance, ApiCallTimeoutException at 95 s, so no replay. One timing gap inline.

assertEquals("preserved", query(port, "SELECT value FROM upgrade_probe"));
} finally {
client.deleteDBInstance(b -> b.dbInstanceIdentifier(id).skipFinalSnapshot(true));
cleanupDockerInstance(client, id, createFailure, Duration.ofSeconds(90));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the suite's 2m method default (pom.xml:823), a create that uses its 90 s leaves about 30 s here, and JUnit's timeout then hides the create failure. Budget only the time left, or give this test its own @Timeout?

@AndreKurait AndreKurait Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Sekator778 Fixed in c5441b0: this fixture now has an explicit five-minute method timeout, leaving room for the bounded create and cleanup waits so the original create failure survives. All test sources compiled; five cleanup regressions and full Checkstyle passed.

Give the engine upgrade fixture a five-minute method timeout so its bounded create and cleanup waits preserve the original create failure.

Signed-off-by: Andre Kurait <AndreKurait@gmail.com>
@AndreKurait

Copy link
Copy Markdown
Contributor Author

The shard 4 failure is a separate PyPI retention race: hyphenated sdists and underscored wheels used different locks for the same package version. Fixed in #8201. The new regression fails on the baseline; the fix passes 52 focused tests and the unchanged seven-case real Docker integration suite. Test-owned containers and volumes were cleaned up.

@pgermosen pgermosen added chore rds Amazon Relational Database Service (RDS) labels Nov 13, 2026

@pgermosen pgermosen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AndreKurait thanks, the single-attempt client and the bounded cleanup read well, and the five-minute timeout leaves room for both waits.

One thing in RdsDockerCleanupTest: toleratesAbsentInstanceOnlyAfterServerRejection passes a mocked RdsException. The suite's subclass mock maker never runs Throwable's constructor, so addSuppressed on that mock records nothing and getSuppressed() stays empty whatever cleanup does. With the DbInstanceNotFoundException catch removed from cleanupDockerInstance, the test still passes. A real exception, RdsException.builder().message("rejected").statusCode(400).build() as absent() builds its own, would make that assertion count.

@pgermosen pgermosen added the waiting-author Review posted; waiting on the PR author to respond label Nov 13, 2026
Construct the rejection exception so the absent-instance cleanup assertion detects an incorrectly suppressed deletion failure.

Signed-off-by: Andre Kurait <akurait@amazon.com>
@AndreKurait

Copy link
Copy Markdown
Contributor Author

@pgermosen Fixed in 08a4fd5. The rejection now uses a real RdsException. With only the not-found catch removed, the previous mocked test still passes and the corrected test fails on the suppressed error. Restored the helper and verified all five cleanup tests, package, full Checkstyle and static gates. No production or Docker lifecycle changes.

@pgermosen pgermosen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AndreKurait thanks, the rejection test now throws a real RdsException, so its empty suppressed list only holds while the not-found catch is there. Good to go.

@pgermosen pgermosen removed the waiting-author Review posted; waiting on the PR author to respond label Nov 13, 2026
@pgermosen
pgermosen merged commit f64ca5e into floci-io:main Nov 13, 2026
23 checks passed
@hectorvent

Copy link
Copy Markdown
Member

🎉 This PR is included in version 2.5.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore rds Amazon Relational Database Service (RDS) released

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants