Repository navigation
feat(elasticache): add an explicit shell-less backend path for Redis-compatible images - #8642
Conversation
|
🎉 Thanks for your first pull request to Floci! Your CI checks need a maintainer to approve them before they run. That is GitHub's standard gate on first-time contributors, not a problem with your PR — so if the checks look like they are doing nothing, that is why. Once a maintainer approves, CI and the compatibility suite start automatically. Nothing is needed from you in the meantime. While you wait, a couple of things that make review faster:
Come join us in Slack — it is the fastest way to reach maintainers if you get stuck, or want feedback on an approach before investing more time in it. |
PR Summary by QodoAdd an explicit shell-less backend path for ElastiCache
AI Description
Diagram
High-Level Assessment
Files changed (16)
|
Code Review by Qodo
1. Remote clients can bypass cache auth
|
| List<String> arguments = new ArrayList<>(List.of("--port", String.valueOf(BACKEND_PORT), | ||
| "--protected-mode", "no", "--loglevel", "verbose")); |
There was a problem hiding this comment.
1. Remote clients can bypass cache auth 🐞 Bug ⛨ Security
start passes --protected-mode no to shell-less backends while native-mode Docker publishing leaves the backend port bound to the host's interfaces. Before proxy-driven backend ACL initialization, anyone who can reach that published port can send cache commands without passing through the group's token, IAM, or user-access checks.
Agent Prompt
### Issue description
Shell-less mode disables backend protected mode, but native-mode Docker publishes the unauthenticated backend port without a host-IP restriction. Clients reaching that port bypass the ElastiCache proxy.
### Fix Focus Areas
- src/main/java/io/floci/aws/services/elasticache/container/ElastiCacheContainerManager.java[190-213]
- src/main/java/io/floci/aws/core/common/docker/ContainerLifecycleManager.java[1813-1821]
### Recommended Fix
Restrict native backend bindings to an interface reachable by Floci but not remote clients, such as loopback for a local daemon. For remote-daemon configurations, provide an isolated backend transport or initialize backend authentication before making a port reachable; test that a remote client cannot connect directly.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| try (ValkeyClusterFormation.RespClient client = new ValkeyClusterFormation.RespClient( | ||
| handle.getHost(), handle.getPort(), SNAPSHOT_SAVE_TIMEOUT_MS)) { | ||
| Object reply = client.call("SAVE"); |
There was a problem hiding this comment.
2. Cache users prevent shell-less snapshots 🐞 Bug ≡ Correctness
snapshotWithoutShell opens a fresh backend connection and sends SAVE without authenticating it. After a user-group client authenticates through the proxy, backend ACL initialization disables the default user, so the direct command receives NOAUTH and CreateSnapshot fails.
Agent Prompt
### Issue description
The shell-less snapshot client cannot issue SAVE or CONFIG GET once proxy-driven ACL initialization disables the backend default user.
### Fix Focus Areas
- src/main/java/io/floci/aws/services/elasticache/container/ElastiCacheContainerManager.java[401-416]
- src/main/java/io/floci/aws/services/elasticache/proxy/ElastiCacheBackendAcl.java[26-55]
### Recommended Fix
Give the snapshot operation a narrowly scoped way to authenticate with the proxy-managed backend administrator before issuing SAVE and CONFIG GET. Cover a snapshot taken after a user-group client has authenticated.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
pgermosen
left a comment
There was a problem hiding this comment.
@murtazayusuf welcome, and thanks for your first PR! It follows the plan agreed in #8209: an explicit setting, flags as arguments, and restore by placing the file before start, with no DEBUG.
A couple of small things. The commit isn't signed, which main requires; could you sign it and force-push (GitHub's guide)? And verifySnapshotDirectory checks dir but not dbfilename, so a server loading another file name starts empty yet looks restored; could it compare the name when the server reports one?
It would also help to share containerOf, docker and the RESP helpers between the two new integration tests through ElastiCacheTestConnections, and to fold createAndStart(spec, beforeStart) into the existing private createAndStart.
…compatible images An image with no shell and no CLI cannot take VALKEY_EXTRA_FLAGS or the valkey-cli based snapshot export and restore. floci.services.elasticache.shell-less-backend (off by default, never inferred from the image name) selects another path: server flags are container arguments, a snapshot is taken with SAVE and CONFIG GET dir/dbfilename over the RESP connection and read out of the container through the Docker archive API, and a snapshot is restored by placing its file in the data directory before the server starts, so DEBUG is not enabled anywhere. A failed restore removes only the new container. The default valkey/valkey path is unchanged. After a restore the server's dir and, when it reports one, its dbfilename are compared with where the file was placed, so a server that loads another file cannot look restored. The docs list what such an image has to support.
aa517f0 to
08d1aeb
Compare
|
Thanks @pgermosen. All done in 08d1aeb, one signed commit, force-pushed as you asked:
On the two Qodo findings I did not change in code:
The docs section also lists what an image has to support now, ACL users with access strings included. Same test selection as before: 1304 run, 0 failures, 3 skipped (the GPU and platform Docker tests); checkstyle and |
pgermosen
left a comment
There was a problem hiding this comment.
@murtazayusuf thanks, all of it is in: the commit is signed, a server loading another file name now fails the restore and names the setting to change, the two integration tests share ElastiCacheTestConnections, and createAndStart takes the callback in one place. Your reasoning on protected mode holds too, since the default image runs the same way on the same published port.
(follow-up, separate PR) Giving the snapshot connection credentials once the proxy has turned the backend's default user off, on both paths, as you suggested.
|
Thanks @pgermosen and @hectorvent for the quick reviews, and @AndreKurait for checking the plan before I started; it made this an easy one to land. I'll open the follow-up for the snapshot connection's credentials (both paths) as its own PR. |
Summary
Closes #8209
ElastiCache starts its Redis/Valkey backend through the image's
VALKEY_EXTRA_FLAGShook and takes and restores snapshots withvalkey-cli/redis-clirun inside the container (exec,DEBUG RELOAD). That only works for images built likevalkey/valkey. An image with no shell and no CLI cannot be used as a backend, so this adds a second path for those, as agreed in the issue.The path is chosen by a new setting,
floci.services.elasticache.shell-less-backend(FLOCI_SERVICES_ELASTICACHE_SHELL_LESS_BACKEND). It isfalseby default and is never inferred from the image name, so which path runs is always the one the user configured. With the default, nothing changes:valkey/valkeystill getsVALKEY_EXTRA_FLAGSand the CLI path, with--enable-debug-command local.With
shell-less-backend: true:--port 6379 --protected-mode no --loglevel verbose, plus the cluster-mode flags for a cluster-mode group) instead ofVALKEY_EXTRA_FLAGS. The image's own default arguments are replaced, so the port and protected mode are stated explicitly.CreateSnapshotrunsSAVEandCONFIG GET dir/CONFIG GET dbfilenameover the existing RESP connection to the backend, then reads the file out of the container with the Docker archive API. Nothing is executed inside the container.CreateReplicationGroupwithSnapshotNamecreates the container, puts the snapshot file in its data directory (the image's working directory, where a server started without--dirkeeps its data), then starts it, so the server loads the file itself. There is noDEBUG RELOAD, andDEBUGis not enabled anywhere new. After start, Floci checks over RESP that the server'sdiris the directory the file was placed in, so a server that reads somewhere else cannot look like a successful restore.InvalidSnapshotState. The replication group the snapshot came from and the snapshot are not touched.A second setting,
floci.services.elasticache.shell-less-backend-dbfilename(defaultdump.rdb), names the file the server loads at startup. A restore has to place the file before the server runs, so it cannot ask the server; a snapshot export uses what the server reports fordbfilenameand falls back to this setting when the server reports nothing. KiviDB, for example, keepsdump.kdband reports an emptydbfilename.MemoryDB is not touched: it has its own container manager, which does not use any of the changed classes (it only shares
RespLineReader, unchanged).Small supporting changes:
ContainerLifecycleManager.createAndStart(spec, beforeStart)(create, run a callback, start; removes the container if either step fails), and a read timeout argument on the package-privateValkeyClusterFormation.RespClient, soSAVEcan wait longer than the 10 s default.Docs: the setting is described in
docs/services/elasticache.md(new section "Images with no shell or CLI", configuration table, snapshots paragraph),docs/configuration/environment-variables.mdanddocs/configuration/advanced/application-yml.md.Type of change
fix:)feat:)feat!:orfix!:)AWS Compatibility
No change to the ElastiCache wire protocol, request or response shapes: the setting only changes how the backend container is started and how its data is read and loaded.
CreateSnapshotandCreateReplicationGroupwithSnapshotNameanswer the same as before.Testing
Run on an Ubuntu VM (Docker, Temurin/OpenJDK 25).
ElastiCacheShellLessContainerManagerTest(unit): flags as container arguments, normal and cluster mode; noVALKEY_EXTRA_FLAGSand no--enable-debug-commandwith the setting on; the default path still usesVALKEY_EXTRA_FLAGSand no arguments; a startup seed is refused with the setting off.ElastiCacheShellLessSnapshotServiceTest(unit, container layer mocked): the snapshot goes to the start and never torestore; a start the snapshot cannot be loaded in rolls back only the new group and leaves the source group and snapshot; no daemon leaves nothing behind.ElastiCacheShellLessBackendIntegrationTest(Docker-gated, skipped without a daemon) against the public KiviDB imageghcr.io/kividbio/kividb:v1.2.2, pinned by digestsha256:8853c7b1e73e604502fd584533440cb1205ae311fe805d520543b4e85a4b71e9, a distroless image with no shell: the test first checks thatshcannot be executed in it; arguments delivered; data through the proxy; snapshot export and restore (data written after the snapshot is absent from the restored group,--enable-debug-commandnot passed); group auth token enforced by the proxy; a user with an access string (on ~app:* -@all +get +set +ping) is refusedSET other:keyandFLUSHALLwith the backend's-NOPERMreaching the client; a snapshot the server cannot load fails the start and leaves no container, source group and snapshot intact; deleting groups removes their containers.ElastiCacheShellLessClusterIntegrationTest(Docker-gated): cluster-mode flags arrive as container arguments on every node, the cluster forms, slots are served andMOVEDis returned, delete removes every node container. This one usesvalkey/valkey:8with the setting on, because KiviDB v1.2.2 does not accept--cluster-announce-client-ipv4, which Floci passes to every cluster-mode node.Results on the VM (Ubuntu 24.04, OpenJDK 25, Docker 29):
./mvnw test -Dtest="io.floci.aws.services.elasticache.**.*Test,*ElastiCache*Test,io.floci.aws.services.memorydb.**.*Test,io.floci.aws.core.common.docker.**.*Test"gaveTests run: 1300, Failures: 0, Errors: 0, Skipped: 3, BUILD SUCCESS. The 3 skips are the GPU and platform Docker tests incore.common.docker, which need hardware or an emulated platform. All the new tests above ran and passed, as didElastiCacheIntegrationTest,ElastiCacheClusterIntegrationTestandElastiCacheAclProxyIntegrationTeston the default image../mvnw checkstyle:check,make docs-checkandmake partition-checkpass. A single./mvnw testover the whole suite did not finish on a 4-vCPU, 16 GB machine: the Quarkus build ran out of heap (-Xmx6g) before the first reports, so CI's five shards remain the check for the rest.Checklist