Skip to content

fix(emr-serverless): queue job runs beyond an application's maxConcurrentRuns - #9124

Merged
pgermosen merged 1 commit into
mainfrom
fix/emr-serverless-run-queuing
Nov 21, 2026
Merged

pgermosen merged 1 commit into
mainfrom
fix/emr-serverless-run-queuing

Conversation

@pgermosen

@pgermosen pgermosen commented Nov 21, 2026 •

Copy link
Copy Markdown
Member

Summary

An application's schedulerConfiguration was stored and echoed, but no run was ever QUEUED: every run started at once, and emr-serverless.md listed queuing as not emulated.

Runs now queue, in either runner mode:

  • a run beyond the application's maxConcurrentRuns is QUEUED, and the first queued run starts the moment a running one ends or is cancelled;
  • a run still queued after queueTimeoutMinutes is CANCELLED;
  • a run that leaves the queue reports queuedDurationMilliseconds;
  • a queued run is cancelled at once, keeps StopApplication from stopping the application, and in the docker runner mode has its container started only once it leaves the queue.

The configuration itself:

  • queuing applies from release 7.0.0, and from 7.3.0 it is on by default: an application that sets no limits runs with the defaults, 15 runs and 360 minutes. The AWS runtime for Apache Spark's releases count as later than 7.3.0;
  • a member a request leaves out takes its default, or keeps the value already set; an empty schedulerConfiguration turns queuing off, on a release that queues by default too, and the application then returns none;
  • maxConcurrentRuns outside 1 to 1000 and queueTimeoutMinutes outside 15 to 720 are refused with ValidationException;
  • a STARTED application takes a new maxConcurrentRuns, and a higher one starts queued runs at once; any other scheduler change still needs it stopped.

This is the second of two parts; the first, EMR on EKS's concurrent job limits, is #9120. queuedDurationMilliseconds was listed with other unreported members under a later item; it comes here, as only a queue gives it a value.

Type of change

  • Bug fix (fix:)
  • New feature (feat:)
  • Breaking change (feat!: or fix!:)
  • Docs / chore

AWS Compatibility

The incorrect behavior: the limits were stored and never applied, and no run was ever QUEUED.

  • Queuing: the user guide's Job concurrency and queuing for an EMR Serverless application: "if your job run concurrency is 10, only ten jobs are run at a time on your application. Remaining jobs are queued until one of the running jobs terminates. If queue timeout is reached earlier, your job times out." Its considerations give the releases ("supported on Amazon EMR release 7.0.0 and higher", "enabled by default on Amazon EMR release 7.3.0 and higher"), the ranges ("1 to 1000", "15 to 720"), "You can update maxConcurrentRuns for an application in the STARTED state", and that it "applies to batch and streaming jobs" and not interactive ones. The model's SchedulerConfiguration gives the defaults, 15 and 360 minutes.
  • A queue timeout: the guide's job run states make "the job run has timed out" a reason to be CANCELLING, so a run that times out in the queue is CANCELLED. Nothing records its stateDetails, so that is Floci's wording.
  • How the configuration is kept: Terraform's acceptance tests, run against AWS in hashicorp/terraform-provider-aws#44589, found that "specifying an empty SchedulerConfiguration{} (without any attributes) disables the feature", after which the application has none, and that "if only one of the two parameters (MaxConcurrentRuns or QueueTimeoutMinutes) is provided, the other parameter is automatically set to its default value". Those tests started from no configuration; where one is set, Floci keeps the other member's value rather than resetting it.

Not recorded anywhere, and documented in emr-serverless.md as Floci's reading:

  • Whether GetApplication shows the defaults for an application on 7.3.0 or later that never set any. Floci applies them without showing them.
  • The wording of the out-of-range refusals.

The account's limit of 2000 QUEUED runs is not enforced.

Checklist

  • ./mvnw test passes locally
  • New or updated integration test added
  • Commit messages follow Conventional Commits
  • I have no more than 2 open, non-draft pull requests and no more than 4 open pull requests in total (drafts included) in this repository (maintainers and dependency bots are exempt).

On the first box: the whole suite wasn't run. On the last: maintainers are exempt from the limit.

EmrServerlessJobRunServiceTest:

  • aRunBeyondMaxConcurrentRunsIsQueuedAndStartsTheMomentARunEnds: with a limit of 1 the second run is QUEUED, listed under QUEUED, and starts when the first ends, with its queuedDurationMilliseconds;
  • aRunStillQueuedAtTheQueueTimeoutIsCancelled: CANCELLED at createdAt plus the timeout, never started;
  • aCancelOrAHigherLimitOnTheStartedApplicationStartsTheQueuedRunsInOrderAtOnce: a queued run is cancelled at once, a cancelled running run's slot goes to the next queued one, StopApplication is refused while runs wait, and a raised limit starts the rest;
  • runsQueueByDefaultFromRelease730AndNeverBeforeRelease700: 7.3.0 and emr-spark-8.0.0 queue the 16th run with no configuration, 7.2.0 does not, and 6.15.0 does not even with one;
  • aQueuedContainerRunStartsItsDriverOnlyOnceTheRunningOneHasEnded: in the docker mode the queued run's driver is launched only after the first run ends, and a queued run has no driver to stop;
  • aQueuedBatchRunThatStartsWithoutADurationEndsAsItStarts: with the default duration, a batch run leaving the queue is SUCCESS at once;
  • manyRunsEndingTogetherHandTheirSlotsOnInOrder: 20 runs on five slots replayed in one read, each queued run starting as the run five places ahead ends;
  • anEmptySchedulerConfigurationTurnsQueuingOffOnAReleaseThatQueuesByDefault;
  • aQueuedRunWhoseDriverIsNotItsApplicationsFailsAsItLeavesTheQueueAndTakesNoSlot: a hive run queued on a Spark application fails as it leaves the queue, and the valid run behind it starts.

EmrServerlessServiceTest:

  • aSchedulerConfigurationFillsWhatItLeavesOutAndAnEmptyOneRemovesIt: missing members filled, an empty one removed, out-of-range and non-integer values refused;
  • aStartedApplicationTakesANewMaxConcurrentRunsButNoOtherSchedulerChange;
  • anUpdateRefusedForItsSchedulerConfigurationChangesNothingElse: a refused scheduler value leaves the capacity sent beside it unapplied.

Settling an application replays its runs from a priority queue of endings and a deque of queued runs, reads that application's runs alone through an index of run ids by application, and stores what it changed in one write.

EmrServerlessJobRunIntegrationTest.aRunBeyondMaxConcurrentRunsWaitsQueuedUntilTheRunningOneIsCancelled: over the wire, the filled configuration, a QUEUED streaming run listed by state, started once the running one is cancelled, and an empty configuration removed. Every EMR Serverless test class but the Docker and nightly ones passes as a whole. Checkstyle, docs-check and partition-check are clean.

@pgermosen pgermosen added bug Something isn't working emr Amazon EMR (Elastic MapReduce) labels Nov 21, 2026
@qodo-code-review

qodo-code-review Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

PR Summary by Qodo

Queue EMR Serverless runs at the application concurrency limit

🐞 Bug fix 🧪 Tests 📝 Documentation 🕐 40+ Minutes

Grey Divider

AI Description

• Enforce release-aware concurrency limits and queue timeouts instead of starting every job run
 immediately.
• Start queued runs when capacity opens; report queue duration and support immediate queued-run
 cancellation.
• Validate scheduler settings, allow live concurrency increases, and cover both runner modes with
 tests.
Merge Risk 🟡 Medium

Blast radius 🟡 Medium 💥 2 capabilities

  • 2 affected capabilities within one bounded functional scope: emr_serverless_job_scheduling.
  • The changed admission, replay, persistence, API reads, cancellation, application stopping, and container launch all implement one end-to-end outcome: deciding when an application's job run executes and what lifecycle state the API reports.
  • These are implementation hops of the same bounded scheduling integration, not separate capabilities.
  • Create/update requests, stored Application scheduler fields, API responses, and subsequent job-run limit evaluation form one bounded application scheduling configuration behavior.
  • The controller and model changes are contract and persistence hops of that behavior, while the job-run consumer is the downstream execution hop.
Behavioral complexity 🔴 High 🧩 Multiple interactions

  • Before: each run was settled independently using its own state, duration, and container thread.
  • After: all runs for an application are collected, sorted, replayed through timed completions and queue timeouts, and then persisted before newly eligible container runs are launched.
  • New correctness conditions: (1) active-slot count must agree with persisted nonterminal states and executing threads; (2) queue order must remain stable while completions, cancellations, updates, and timeouts interleave; (3) a slot must be handed to exactly one queued run; (4) persistence must precede launch so a concurrent read or shutdown cannot observe an untracked driver; and (5) immediate zero-duration runs must release their slot again during the same replay.
  • These are newly interacting lifecycle states and coordinated persistence/launch invariants, not merely reuse of existing state helpers.
Potential damage 🟡 Medium 🩹 Recoverable

  • A wrong queue decision can delay a job, cancel it at the wrong time, start an unexpected job, or interrupt a container workload.
  • The affected capability is job execution and lifecycle reporting, not arbitrary unrelated application data.
  • Reverting the code prevents new faulty decisions but does not automatically repair already persisted QUEUED, CANCELLED, RUNNING, or FAILED run records; operators may need to cancel/resubmit affected runs or correct the application configuration.
  • No external financial, confidentiality, or irreversible data effect is established by this source.
  • Usually recoverable by code rollback plus cancelling/resubmitting affected runs or waiting for a correct subsequent settlement.
  • Existing persisted run states and any external work already launched are not automatically rolled back; manual reconciliation may be required for a run that was incorrectly started or cancelled.
Assessed commit a5e0e0f
Diagram

graph TD
  A["Scheduler limits"] --> B{"Slot available?"} -->|Yes| C["Run active"] --> F["Run ends"]
  H["Job submitted"] --> B
  B -->|No| D["FIFO queue"] --> E{"Queue outcome?"} -->|Timeout or cancel| G["Run cancelled"]
  F --> E
  E -->|Slot opens| C
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Background scheduler with timeout tasks
  • ➕ Could expire queued runs without waiting for an API call.
  • ➕ Could make queue transitions independent of run reads.
  • ➖ Requires durable timer recovery and more concurrency coordination.
  • ➖ Adds infrastructure to a service that already settles immediate-mode runs lazily.

Recommendation: Keep the PR's application-wide settlement approach: it fits the existing immediate-run lifecycle, replays transitions at their intended times, and launches Docker runs when slots open. A background scheduler is worth considering only if timeouts must become visible without any subsequent request.

Files changed (9) +566 / -48

Bug fix (5) +311 / -43
EmrServerlessController.javaRoute application updates through run coordination +1/-1

Route application updates through run coordination

• Sends UpdateApplication through the job-run service so queued runs are settled before and after scheduler-limit changes.

src/main/java/io/floci/aws/services/emrserverless/EmrServerlessController.java

EmrServerlessJobRunService.javaEnforce queue-aware job-run lifecycles +186/-35

Enforce queue-aware job-run lifecycles

• Admits excess runs as QUEUED and settles active runs, queue timeouts, cancellations, and FIFO slot handoffs at their effective times. Defers Docker launches until a queued run starts and records its queue duration.

src/main/java/io/floci/aws/services/emrserverless/EmrServerlessJobRunService.java

EmrServerlessScheduler.javaDefine effective scheduler limits and configuration rules +101/-0

Define effective scheduler limits and configuration rules

• Adds release-aware concurrency and timeout defaults, configuration merging, range validation, and restrictions on scheduler updates to STARTED applications.

src/main/java/io/floci/aws/services/emrserverless/EmrServerlessScheduler.java

EmrServerlessService.javaApply and validate scheduler configuration +17/-7

Apply and validate scheduler configuration

• Merges scheduler settings on create and update, accepts maxConcurrentRuns-only updates while STARTED, and exposes live application lookup for run settlement.

src/main/java/io/floci/aws/services/emrserverless/EmrServerlessService.java

JobRun.javaExpose queued run duration +6/-0

Expose queued run duration

• Adds queuedDurationMilliseconds to job-run responses for runs that have left the queue.

src/main/java/io/floci/aws/services/emrserverless/model/JobRun.java

Tests (3) +228 / -0
EmrServerlessJobRunIntegrationTest.javaExercise queue behavior through the API +29/-0

Exercise queue behavior through the API

• Checks queued state and listing, promotion after cancellation, reported queue duration, and clearing scheduler configuration through HTTP requests.

src/test/java/io/floci/aws/services/emrserverless/EmrServerlessJobRunIntegrationTest.java

EmrServerlessJobRunServiceTest.javaCover queue timing, defaults, and runner modes +135/-0

Cover queue timing, defaults, and runner modes

• Tests FIFO promotion, queue expiration and cancellation, live limit increases, release-dependent defaults, and deferred Docker driver launches.

src/test/java/io/floci/aws/services/emrserverless/EmrServerlessJobRunServiceTest.java

EmrServerlessServiceTest.javaTest scheduler persistence and update validation +64/-0

Test scheduler persistence and update validation

• Verifies default-filled and retained members, empty-configuration removal, invalid-value rejection, and the scheduler updates permitted while an application is STARTED.

src/test/java/io/floci/aws/services/emrserverless/EmrServerlessServiceTest.java

Documentation (1) +27 / -5
emr-serverless.mdDocument concurrency and queue behavior +27/-5

Document concurrency and queue behavior

• Explains release-dependent defaults, configuration rules, queue lifecycle, timeouts, and live concurrency updates. Clarifies that queued Docker runs have no container and removes queuing from the unsupported-features list.

docs/services/emr-serverless.md

@qodo-code-review

qodo-code-review Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Action required

1. Rejected updates can still change capacity ✓ Resolved

Resolving commit: a5e0e0f

Resolution reason
The merged scheduler configuration is computed and validated up front, before any application field
is mutated; the later apply step only stores the already-validated value. A test confirms a rejected
update combined with a capacity change leaves maximumCapacity unchanged.
Description
updateApplication validates the new scheduler configuration in merged after mutating earlier
application settings, including maximumCapacity. When a request supplies both a capacity change
and an out-of-range scheduler value, it returns ValidationException but the earlier mutation
remains on the stored application object.

2. Queued batch runs remain running too long ✓ Resolved

Resolving commit: a5e0e0f

Resolution reason
The post-finding commit replaces the separate post-loop startQueued call with a Replay class whose
run(now) loop is a single while(true) that interleaves timed endings, queue timeouts, and
startQueued(at). startQueued calls running(record) for each immediate-mode run it starts, which
registers that run in the timed PriorityQueue (timedEnding returns its end at the start instant when
the duration is zero). The loop then picks up that ending at the same instant and ends the run as
SUCCESS before the changes are stored. The test
aQueuedBatchRunThatStartsWithoutADurationEndsAsItStarts covers exactly this case.
Description
settleApplication calls startQueued after its replay loop but never checks the newly started
runs for an ending at that same instant. When cancellation or a higher limit opens a slot for an
immediate-mode batch run with the default zero-second duration, that run remains RUNNING until
another request settles the application.


Remediation recommended

3. Large queues slow settlement quadratically ✓ Resolved

Resolving commit: a5e0e0f

Resolution reason
The settlement was refactored into a Replay class. Timed endings are now held in a PriorityQueue
ordered by end time (timed), and the queue is a java.util.ArrayDeque consumed with pollFirst
(queue), so each ending and promotion no longer rescans the active list or shifts an ArrayList
front. Replay is still chronological and FIFO. A test
(manyRunsEndingTogetherHandTheirSlotsOnInOrder) covers a 20-run simultaneous backlog. Also, runsOf
now reads only the application's run ids instead of scanning the whole store each time.
Description
settleApplication rescans every active run for each ending and removes queued runs from the front
of an ArrayList for each promotion. When many runs finish or leave a large queue together, these
repeated operations are quadratic and hold the synchronized service lock throughout.

4. Every job run poll now scans the whole store ✓ Resolved

Resolving commit: a5e0e0f

Resolution reason
Replaced the full store scan in settleApplication with runsOf(), which reads an application-scoped
index of run ids (runIdsByApplication, built once per account from store.scan and kept current by
indexRun on new runs, cleared on reset) and fetches only that application's records via store.get.
Also restructured the replay into a Replay class with a timed priority queue and a deque so the
replay work is proportional to the application's runs. This is an application-scoped index approach
similar in spirit to the suggested fix, though it does not skip replay when no runs are queued; the
full scan now happens only once per account for index construction.
Description
settle(record) now delegates to settleApplication, whose unrestricted store scan traverses every
run in the account before selecting one application's runs. GetJobRun, the existing-client-token and
second-settlement paths in StartJobRun, CancelJobRun, dashboard reads, and container-run ended()
callbacks incur that traversal under the service-wide synchronized lock, turning a single-run read
from O(1) settlement into O(account runs).

5. Empty scheduler config fails to disable queuing on 7.3+ ✓ Resolved

Resolving commit: a5e0e0f

Resolution reason
Added Application.schedulerOff, set by EmrServerlessScheduler.apply when merged config is null
(empty request), and checked first in EmrServerlessScheduler.limits to return null (no queuing
defaults). Service update/create route through apply, docs and tests were updated to show an empty
configuration turns queuing off on 7.3+ releases.
Description
EmrServerlessScheduler.limits treats a null or empty configuration on 7.3.0+ (and on emr-spark) as
the 15/360 defaults, so the null that merged returns for {} turns queuing straight back on. A
user on a current release who sends schedulerConfiguration: {} to turn queuing off still has runs
beyond 15 QUEUED and cancelled after 360 minutes. That goes against the Terraform finding the PR
cites, that an empty configuration "disables the feature".


Informational

6. Queued run that fails at launch ends before it left the queue ✓ Resolved

Resolving commit: a5e0e0f

Resolution reason
The replay now handles a promoted run whose driver mismatches inside startQueued: it calls
leaveQueue(run, at), then ends it as FAILED at the moment it left the queue (at), and takes no slot
(no untimed++), so the loop continues and the next queued run is promoted in the same settle. Also
the mismatch check was moved into a shared driverIsTheApplications helper used by both launch and
startQueued. Note the fix is in the replay rather than in launch; the failing run no longer ends at
createdAt, and the freed slot is refilled within the same settle. A regression test
(aQueuedRunWhoseDriverIsNotItsApplicationsFailsAsItLeavesTheQueueAndTakesNoSlot) is added.
Description
launch ends a driver/application type mismatch at run.getCreatedAt(), while settleApplication
does not refill the slot reserved for a run that fails during launch. When startQueued promotes an
incompatible queued run, it reports an end time earlier than its departure from the queue, and the
next queued run waits for another settle despite the freed slot, potentially reaching its queue
timeout first.

Grey Divider

Context sources
✅ Linked PRs:
  🌐 PR #9120
❌ Tickets: failed to load — Ticket collection could not establish complete evidence
✅ Compliance rules (platform): 21 rules
✅ Web pages:
  +2 more
✅ Cross-repo context — repo relationships
  Explored: repo: floci-io/testcontainers-floci-go (sha: 8db9d48e) — View relationship
  Explored: repo: floci-io/testcontainers-floci-python (sha: e6b79f58) — View relationship
  Explored: repo: floci-io/testcontainers-floci (sha: e900a79e) — View relationship
  Explored: repo: floci-io/testcontainers-floci-node (sha: 1251b1d3) — View relationship
  Explored: repo: floci-io/testcontainers-floci-dotnet (sha: a011afd5) — View relationship
✅ Context preparation
  Past issues: summarized; full text retained for this review
Review mode: Auto: 🧠 Deep: Dense concurrency and lifecycle logic spans multiple independently risky code paths.

Grey Divider

Tip of the day
💡 Did you know, you can add REVIEW.md to your repo root and Qodo follows it on every PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

…rentRuns

An application's schedulerConfiguration was stored and echoed, and no
run was ever QUEUED: every run started at once. The user guide's "Job
concurrency and queuing for an EMR Serverless application" has only
maxConcurrentRuns runs going at a time, "Remaining jobs are queued
until one of the running jobs terminates. If queue timeout is reached
earlier, your job times out."

A run beyond the limit is now QUEUED, in either runner mode, and the
first queued run starts the moment a running one ends or is cancelled.
A run still queued after queueTimeoutMinutes is CANCELLED, as the job
run states give a run that "has timed out" as a reason to be cancelled;
its stateDetails is Floci's wording. A run that leaves the queue
reports queuedDurationMilliseconds. A queued run is cancelled at once,
and keeps StopApplication from stopping the application. In the docker
runner mode a queued run's container starts only once it leaves the
queue.

Queuing is "supported on Amazon EMR release 7.0.0 and higher" and
"enabled by default on Amazon EMR release 7.3.0 and higher", the AWS
runtime for Apache Spark's releases included; an application on 7.3.0
or later that sets no limits runs with the defaults, 15 runs and 360
minutes. Terraform's acceptance tests, run against AWS, recorded how
the configuration is kept: a member a request leaves out takes its
default, and here keeps the value already set where there is one; an
empty one turns queuing off, and the application then has none.
maxConcurrentRuns outside 1 to 1000 and queueTimeoutMinutes outside 15
to 720 are refused with ValidationException. A STARTED application
takes a new maxConcurrentRuns, as "You can update maxConcurrentRuns for
an application in the STARTED state", and a higher one starts queued
runs at once; any other scheduler change still needs it stopped.
@pgermosen
pgermosen force-pushed the fix/emr-serverless-run-queuing branch from bf543cd to a5e0e0f Compare November 21, 2026 01:53
@pgermosen
pgermosen merged commit a87e8e4 into main Nov 21, 2026
25 of 26 checks passed
@pgermosen
pgermosen deleted the fix/emr-serverless-run-queuing branch November 21, 2026 02:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working emr Amazon EMR (Elastic MapReduce)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant