Skip to content

fix(autoscaling): register instances with a Classic Load Balancer on attach and deregister them on detach - #8496

Merged
pgermosen merged 3 commits into
floci-io:mainfrom
mrfelek:fix/autoscaling-attach-classic-load-balancers
Nov 15, 2026
Merged

pgermosen merged 3 commits into
floci-io:mainfrom
mrfelek:fix/autoscaling-attach-classic-load-balancers

Conversation

@mrfelek

@mrfelek mrfelek commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #8469, from its review: AttachLoadBalancers and DetachLoadBalancers only recorded the Classic Load Balancer's name. The group's instances joined a Classic Load Balancer only as the reconciler brought each into service, so those already in service stayed out of one attached later, and every instance stayed in one detached from its group. AWS registers the group's running instances on attach and deregisters them on detach. Classic Load Balancers attached as elb traffic sources were only recorded too.

Both operations now go through changeAttachments, the path #8469 gave the target group calls:

  • attachedClassicLoadBalancers gives the Classic Load Balancers a group has attached: by name, and as elb traffic sources. It sits beside attachedTargetGroups (target groups and elbv2 sources), and both read through one helper.
  • Under the Region's lock and then the group's, the group's instances are deregistered from the target groups and then the Classic Load Balancers the change detached, before the change is stored, in one call to each load balancer. When one refuses, the in-service instances rejoin each load balancer they had already left, and the group keeps everything the change detached, so it is as it was and the detach can be sent again. Before, a refused detach naming two target groups left the first attached without its instances, as Qodo pointed out on this PR. A load balancer already deleted is detached all the same, as deregisterFromClassicLoadBalancers already skips LoadBalancerNotFound.
  • The group's InService instances are then registered with the Classic Load Balancers the change attached. One in Standby or Pending is not; the reconciler registers it as it comes into service.
  • An EKS node group's node is kept in step with its group's Classic Load Balancers as it is with its target groups: it joins them when it is added to the group, and a detach, a deleted group or a detached node takes it out of only those no other group holding it has attached. deregisterFromUnsharedTargetGroups and the new deregisterFromUnsharedClassicLoadBalancers share that look at the Region's other groups.

elb traffic sources now count as Classic Load Balancers wherever instances join or leave one. deregisterFromClassicLoadBalancers(elb, asg, ids), which detaching instances, standby, force delete and every reconciler removal call, reads attachedClassicLoadBalancers. The reconciler's registration of an instance coming into service moves from AutoScalingReconciler to AutoScalingService.registerWithClassicLoadBalancers, beside registerWithTargetGroups, and reads it too.

docs/services/autoscaling.md described AttachLoadBalancers as "stored", DetachLoadBalancers as a detachment only, and the traffic source operations as registering with elbv2 sources only. The four rows now describe what the operations do.

Not in this PR:

  • AttachInstances puts the instances in service without registering them with the group's target groups or Classic Load Balancers, as on main.
  • An EKS group whose delete a load balancer refuses part-way stays, as it does on main when a target group refuses; EksService logs the failure and deletes the node group.

Type of change

  • Bug fix (fix:)
  • New feature (feat:)
  • Breaking change (feat!: or fix!:)
  • Docs / chore

AWS Compatibility

The AttachLoadBalancers reference has "Amazon EC2 Auto Scaling registers the running instances with these Classic Load Balancers", and DetachLoadBalancers has "When you detach a load balancer, it enters the Removing state while deregistering the instances in the group". botocore's Auto Scaling model gives LoadBalancerState's states as "Adding - The Auto Scaling instances are being registered with the load balancer" and "Removing - The Auto Scaling instances are being deregistered from the load balancer".

Checklist

  • ./mvnw test: the 105 test classes the change can affect (the Auto Scaling and EKS classes, and those naming a changed class) pass on Temurin 25 (1,811 tests), with checkstyle; make docs-check passes. This PR's CI runs the full suite.

  • New or updated tests added: AutoScalingClassicLoadBalancerAttachmentTest, on an ordinary group with one InService and one Standby instance, mirroring AutoScalingTargetGroupAttachmentTest:

    • Attaching registers the in-service instance with the load balancer the attach added. It doesn't register the standby instance, or register again with a load balancer already attached.
    • Detaching deregisters the group's instances from the detached load balancer in one call, and the group keeps the one not detached.
    • An elb traffic source registers and deregisters the instances as a Classic Load Balancer does.
    • A force delete deregisters the instances from an elb traffic source as well as from the load balancers attached by name.
    • A refused deregistration leaves the load balancer attached, and the detach succeeds when it is sent again.
    • A Classic Load Balancer deleted before its detach is detached all the same.
    • A detach naming an elbv2 and an elb source, refused by the Classic Load Balancer, puts the in-service instance back in the target group it already left and leaves both attached; sending it again finishes it.
    • Two EKS groups sharing their cluster's node: detaching the load balancer from one leaves the node in it while the other has it attached as an elb source, and detaching it there takes the node out.
    • A node added to an EKS group after a Classic Load Balancer was attached is registered with it.
    • Detaching the node from one EKS group, and deleting the other, takes it out of only the load balancers no remaining group has attached.

    All but the deleted-load-balancer test fail on main. That one passes there, since main sends no deregistration on detach, and guards the new path's handling of a deleted load balancer. AutoScalingGroupFixtures holds the groups this class, AutoScalingTargetGroupAttachmentTest and AutoScalingEksManagedGroupTest share: the ordinary group and its members, and the EKS groups sharing a node. AutoScalingStandbyTest.returningInstanceRetriesFailedRegistrationBeforeCompletingActivity now captures the Classic registration's warning from AutoScalingService, where it is logged since the move. Every Auto Scaling test class and EksNodegroupAutoScalingGroupIntegrationTest pass (809 tests).

  • Commit messages follow Conventional Commits

  • I have no more than 2 open, non-draft pull requests and no more than 4 open pull requests in total (drafts included) in this repository (maintainers and dependency bots are exempt).

…attach and deregister them on detach

AttachLoadBalancers and DetachLoadBalancers only stored the Classic Load
Balancer's name, so the group's instances joined a load balancer only as
they were launched, and stayed in one after it was detached. They now go
through changeAttachments, as the target group calls do since floci-io#8469: the
group's instances leave a detached load balancer before the change is
stored, and its InService instances join an attached one. A refused
deregistration puts back everything the change detached, as a refused
target group does, so sending the detach again finishes it.

elb traffic sources count as Classic Load Balancers throughout, as elbv2
sources count as target groups: for registration as instances come into
service and for deregistration as they leave. The reconciler's Classic
registration moves next to registerWithTargetGroups, so both take the
same path.

An EKS group's node is kept in step with the group's Classic Load
Balancers as with its target groups: it joins them when it is added, and
a detach, a deleted group or a detached node takes it out of only those
no other group holding it has attached.
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Register and deregister Classic ELB instances when ASG attachments change

🐞 Bug fix 🧪 Tests 📝 Documentation 🕐 40+ Minutes

Grey Divider

AI Description

• Register in-service instances when Classic Load Balancers are attached; deregister members on
 detach.
• Treat elb traffic sources as Classic attachments and preserve registrations shared by EKS
 groups.
• Add attachment and failure-retry tests, and update Auto Scaling documentation.
Diagram

graph TD
  A["Attachment APIs"] --> B["Attachment coordinator"] --> C["ASG state"]
  B --> D["Shared-node check"] --> E["Classic ELB"]
  B --> F["ELBv2 targets"]
  B --> E
  G["ASG reconciler"] --> E
  G --> F
Loading
High-Level Assessment

Reusing the existing target-group attachment coordinator is appropriate: it gives Classic ELBs the same locking, detach retry, and EKS sharing behavior without maintaining a separate attachment lifecycle.

Files changed (6) +374 / -77

Bug fix (2) +128 / -67
AutoScalingReconciler.javaUse shared Classic ELB registration during reconciliation +5/-31

Use shared Classic ELB registration during reconciliation

• Replaces the reconciler's private Classic ELB registration method with the Auto Scaling service helper. Instances entering service now register with both named load balancers and 'elb' traffic sources.

src/main/java/io/floci/aws/services/autoscaling/AutoScalingReconciler.java

AutoScalingService.javaSynchronize Classic ELB attachments with group membership +123/-36

Synchronize Classic ELB attachments with group membership

• Routes Classic attachment changes through the existing locked attachment lifecycle, registering in-service members on attach and deregistering members before detach is stored. Includes 'elb' traffic sources in Classic registration paths, restores attachment state after failed deregistration, and protects EKS nodes still shared by another group.

src/main/java/io/floci/aws/services/autoscaling/AutoScalingService.java

Tests (2) +240 / -4
AutoScalingClassicLoadBalancerAttachmentTest.javaTest Classic ELB attachment lifecycle +237/-0

Test Classic ELB attachment lifecycle

• Adds tests for registration, deregistration, 'elb' traffic sources, force deletion, retry after refused deregistration, and already-deleted load balancers. Also covers joining, detaching, and deleting EKS groups that share a node.

src/test/java/io/floci/aws/services/autoscaling/AutoScalingClassicLoadBalancerAttachmentTest.java

AutoScalingStandbyTest.javaCapture standby registration logs from the shared helper +3/-4

Capture standby registration logs from the shared helper

• Changes the standby retry test to capture Classic ELB warnings from AutoScalingService, where registration now occurs.

src/test/java/io/floci/aws/services/autoscaling/AutoScalingStandbyTest.java

Documentation (2) +6 / -6
autoscaling.mdDocument Classic ELB attachment behavior +4/-4

Document Classic ELB attachment behavior

• Explains instance registration and deregistration for named Classic Load Balancers and 'elb' traffic sources, including EKS shared-node behavior. Clarifies that Floci does not run a Classic ELB data plane.

docs/services/autoscaling.md

EksService.javaClarify Classic ELB cleanup on node-group deletion +2/-2

Clarify Classic ELB cleanup on node-group deletion

• Updates the deletion contract comment to cover Classic Load Balancers alongside target groups and note that registrations shared by another node group remain.

src/main/java/io/floci/aws/services/eks/EksService.java

@qodo-code-review

qodo-code-review Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (1) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Action required

1. Failed detaches leave members unregistered ✓ Resolved

Resolving commit: dd061ef

Resolution reason
Detaches are now performed one target group / Classic load balancer at a time, and each completed
deregistration is recorded as a rejoin action. If a later deregistration throws, the rejoin actions
run (re-registering the in-service members with the target groups and load balancers they already
left) before the attachment metadata is restored and the original exception is rethrown. This
addresses the finding's core problem (members left unregistered after a partial failure) with a
different implementation than the suggested per-deregistration tracking plus re-registration, though
it restores via a rejoin list rather than the exact same structure; EKS members are handled via the
unshared-target variants per item.
Description
changeAttachments deregisters target-group members before Classic members, but a later Classic
refusal restores only the attachment collections. When a request detaches both types and
target-group deregistration succeeds first, the failed request leaves the target group attached
without its former members until another operation repairs them.


Informational

2. Attach swallows Classic registration failures, reporting success 📘 Rule violation ☼ Reliability
Description
changeAttachments records a new Classic attachment before registerWithClassicLoadBalancers
catches every registerInstances exception and logs it with warnv when strict is false. When
attachLoadBalancers, attachTrafficSources, or the elb source path adds a load balancer whose
instance is already InService, a non-benign registration failure leaves that instance unregistered
without an error to the caller, and neither a repeated attach nor normal reconciliation revisits it.
Code

src/main/java/io/floci/aws/services/autoscaling/AutoScalingService.java[R1100-1103]

+            } catch (Exception e) {
+                LOG.warnv(e, "ASG {0}: could not register {1} with Classic ELB {2}: {3}",
+                        asg.getAutoScalingGroupName(), instances, loadBalancer, e.getMessage());
+                if (strict && !(e instanceof AwsException aws
Relevance

● Weak

Same swallow-with-strict=false pattern for target groups was rejected and explicitly dismissed as
intentional on the prior PR.

PR-#8469

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The attach path records membership before calling registerWithClassicLoadBalancers with
strict=false (AutoScalingService.java:1740-1745). The helper catches all registration exceptions
on that path (AutoScalingService.java:1088-1109), rather than limiting suppression to
LoadBalancerNotFound; because the attachment is already recorded, a repeated attach has no new
attachment to process, and reconciliation excludes an already attached, in-service instance.

Rule 3865600: Do not swallow exceptions broadly; suppress only known benign errors
src/main/java/io/floci/aws/services/autoscaling/AutoScalingService.java[1088-1109]
src/main/java/io/floci/aws/services/autoscaling/AutoScalingService.java[1743-1745]
src/main/java/io/floci/aws/services/autoscaling/AutoScalingService.java[1740-1745]
src/main/java/io/floci/aws/services/autoscaling/AutoScalingService.java[1088-1107]
src/main/java/io/floci/aws/services/autoscaling/AutoScalingService.java[1544-1551]
src/main/java/io/floci/aws/services/autoscaling/AutoScalingReconciler.java[285-308]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Classic attachment-time registration can fail for existing in-service instances while the attach reports success. Because membership is recorded first, repeating the attach and normal reconciliation do not repair the missing registration.

## Fix Focus Areas
- src/main/java/io/floci/aws/services/autoscaling/AutoScalingService.java[1088-1109]
- src/main/java/io/floci/aws/services/autoscaling/AutoScalingService.java[1740-1745]

## Recommended Fix
Propagate non-benign registration errors on the attach path, either by passing `true` for `strict` or by changing the catch to rethrow those errors for attaches; continue skipping `LoadBalancerNotFound`. Make the failure recoverable by restoring attachment state and compensating completed registrations when the attach fails, or by persisting retry work for missing members if the attachment remains recorded. Preserve the reconciler’s best-effort behavior for newly promoted instances, and confirm the `addEksManagedInstance` path’s intent before changing it. Add a test in which `registerInstances` throws a non-benign `AwsException` during `attachLoadBalancers` and verify propagation and recoverability; also verify that `LoadBalancerNotFound` is skipped.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Dismiss ↗ | View ↗


Grey Divider

Context sources
✅ Linked PRs:
  🌐 PR #8469
✅ Compliance rules (platform): 15 rules
✅ Web pages:
  +2 more
✅ Cross-repo context — repo relationships
  Explored: repo: floci-io/floci-labs (sha: c6c6bb16) — View relationship
  Explored: repo: floci-io/floci-ui (sha: 60a65cac) — View relationship
Review mode: Auto: 🧠 Deep: Multiple lifecycle paths and shared EKS state warrant redundant review.

Grey Divider

Tip of the day
💡 Did you know, you can reply 'qodo' on any finding to push back, ask questions, or dig deeper

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

… refused

A detach takes the group's members out of the target groups and Classic
Load Balancers it names one at a time. When one refuses, the group keeps
them all attached, but the members had already left those processed
before it, so a target group could stay attached without them until the
detach was sent again. The in-service members now rejoin each load
balancer they already left before the failure is returned, so the group
is as it was. From Qodo's review of floci-io#8496.
@mrfelek

mrfelek commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

On Qodo's second finding (attach logs a Classic registration failure and succeeds): that is deliberate, as for target groups in #8469. AWS attaches a load balancer asynchronously: AttachLoadBalancers returns once the attachment is recorded, and the instances are registered as the load balancer moves through Adding to Added, so a registration that fails doesn't fail the call. A load balancer that doesn't exist is skipped in either mode.

@pgermosen pgermosen added autoscaling Amazon EC2 Auto Scaling bug Something isn't working labels Nov 15, 2026
@pgermosen

Copy link
Copy Markdown
Member

@mrfelek thanks, this reads well. Putting the members back in what they already left when a detach is refused closes the partial-failure gap, and logging a failed registration on attach matches the target group path from #8469.

One thing before it can go in: the new test copies its fixtures. The constants, the constructor's group and member come from AutoScalingTargetGroupAttachmentTest, and FIRST, SECOND, NODE, node() and groupsSharingTheNode() from AutoScalingEksManagedGroupTest. Could those move into one fixture the three classes share?

@pgermosen pgermosen added the waiting-author Review posted; waiting on the PR author to respond label Nov 15, 2026
@mrfelek

mrfelek commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, will do: the group with one InService and one Standby member, member, and the EKS groups sharing a node (FIRST, SECOND, NODE, node(), groupsSharingTheNode()) move into one fixture in the autoscaling test package, and the three classes build on it. I'll push it to this PR.

@AndreKurait AndreKurait left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up at 300b0a0: compiled the extracted shared fixtures and directly ran 25 test methods across Classic ELB attachment, target-group attachment, and EKS managed-group tests; all passed. Production is unchanged from my prior dd061ef review. Validation uses mocked ELB dependencies, not full JUnit, HTTP/SDK, or live AWS.

From the review of floci-io#8496. AutoScalingClassicLoadBalancerAttachmentTest
copied its ordinary group, with one member in service and one in standby,
from AutoScalingTargetGroupAttachmentTest, and its EKS groups sharing a
node from AutoScalingEksManagedGroupTest. AutoScalingGroupFixtures now
holds them: the constants, the service, the ordinary group and member,
and the node, its groups' tags and groupsSharingTheNode. The three
classes build on it, with their tests as they were.
@mrfelek

mrfelek commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Done in 300b0a0: AutoScalingGroupFixtures holds the constants, the service, the ordinary group with its InService and Standby members, and the EKS node, its groups' tags and groupsSharingTheNode. AutoScalingTargetGroupAttachmentTest, AutoScalingEksManagedGroupTest and AutoScalingClassicLoadBalancerAttachmentTest build on it, with their tests as they were; every Auto Scaling class and EksNodegroupAutoScalingGroupIntegrationTest pass (809 tests).

The red native arm64 / sdk-test-java on dd061ef is unrelated: RdsControlPlaneTest.cleanup failed deleting its subnet ("The subnet ... has dependencies and cannot be deleted"), in the RDS SDK compatibility tests this PR doesn't touch.

@mrfelek

mrfelek commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

A correction to my note above: native arm64 / sdk-test-java failed the same way again on 300b0a0 (RdsControlPlaneTest.cleanup, its subnet and then its VPC "has dependencies"), and among today's runs of the compatibility workflow it fails only on this branch, so I can't call it unrelated yet. The class runs all its 26 tests and its DB instances are deleted before the cleanup, but the run's log doesn't show what still holds the subnet, and those tests need Docker, which I can't run here. I'm looking into it before asking for anything.

@mrfelek

mrfelek commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Found it: the same RdsControlPlaneTest.cleanup:142 failure ("The subnet ... has dependencies and cannot be deleted") hit native arm64 / sdk-test-java today on #8333 and #8395 too, so it's a flake of that class's cleanup on main rather than this branch. Could you rerun the job? I'll look at the cleanup separately.

@pgermosen pgermosen left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mrfelek thanks, AutoScalingGroupFixtures is what I had in mind: the three classes now build on one ordinary group and one pair of EKS groups, with their tests as they were. Registering on attach, deregistering on detach and putting the members back when a detach is refused all read well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autoscaling Amazon EC2 Auto Scaling bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants