Repository navigation
fix(emr-serverless): queue job runs beyond an application's maxConcurrentRuns - #9124
Conversation
PR Summary by QodoQueue EMR Serverless runs at the application concurrency limit
AI Description
Merge Risk
|
Code Review by Qodo
1.
|
…rentRuns An application's schedulerConfiguration was stored and echoed, and no run was ever QUEUED: every run started at once. The user guide's "Job concurrency and queuing for an EMR Serverless application" has only maxConcurrentRuns runs going at a time, "Remaining jobs are queued until one of the running jobs terminates. If queue timeout is reached earlier, your job times out." A run beyond the limit is now QUEUED, in either runner mode, and the first queued run starts the moment a running one ends or is cancelled. A run still queued after queueTimeoutMinutes is CANCELLED, as the job run states give a run that "has timed out" as a reason to be cancelled; its stateDetails is Floci's wording. A run that leaves the queue reports queuedDurationMilliseconds. A queued run is cancelled at once, and keeps StopApplication from stopping the application. In the docker runner mode a queued run's container starts only once it leaves the queue. Queuing is "supported on Amazon EMR release 7.0.0 and higher" and "enabled by default on Amazon EMR release 7.3.0 and higher", the AWS runtime for Apache Spark's releases included; an application on 7.3.0 or later that sets no limits runs with the defaults, 15 runs and 360 minutes. Terraform's acceptance tests, run against AWS, recorded how the configuration is kept: a member a request leaves out takes its default, and here keeps the value already set where there is one; an empty one turns queuing off, and the application then has none. maxConcurrentRuns outside 1 to 1000 and queueTimeoutMinutes outside 15 to 720 are refused with ValidationException. A STARTED application takes a new maxConcurrentRuns, as "You can update maxConcurrentRuns for an application in the STARTED state", and a higher one starts queued runs at once; any other scheduler change still needs it stopped.
bf543cd to
a5e0e0f
Compare
Summary
An application's
schedulerConfigurationwas stored and echoed, but no run was everQUEUED: every run started at once, andemr-serverless.mdlisted queuing as not emulated.Runs now queue, in either runner mode:
maxConcurrentRunsisQUEUED, and the first queued run starts the moment a running one ends or is cancelled;queueTimeoutMinutesisCANCELLED;queuedDurationMilliseconds;StopApplicationfrom stopping the application, and in thedockerrunner mode has its container started only once it leaves the queue.The configuration itself:
schedulerConfigurationturns queuing off, on a release that queues by default too, and the application then returns none;maxConcurrentRunsoutside 1 to 1000 andqueueTimeoutMinutesoutside 15 to 720 are refused withValidationException;STARTEDapplication takes a newmaxConcurrentRuns, and a higher one starts queued runs at once; any other scheduler change still needs it stopped.This is the second of two parts; the first, EMR on EKS's concurrent job limits, is #9120.
queuedDurationMillisecondswas listed with other unreported members under a later item; it comes here, as only a queue gives it a value.Type of change
fix:)feat:)feat!:orfix!:)AWS Compatibility
The incorrect behavior: the limits were stored and never applied, and no run was ever
QUEUED.SchedulerConfigurationgives the defaults, 15 and 360 minutes.CANCELLING, so a run that times out in the queue isCANCELLED. Nothing records itsstateDetails, so that is Floci's wording.SchedulerConfiguration{}(without any attributes) disables the feature", after which the application has none, and that "if only one of the two parameters (MaxConcurrentRunsorQueueTimeoutMinutes) is provided, the other parameter is automatically set to its default value". Those tests started from no configuration; where one is set, Floci keeps the other member's value rather than resetting it.Not recorded anywhere, and documented in
emr-serverless.mdas Floci's reading:GetApplicationshows the defaults for an application on 7.3.0 or later that never set any. Floci applies them without showing them.The account's limit of 2000
QUEUEDruns is not enforced.Checklist
./mvnw testpasses locallyOn the first box: the whole suite wasn't run. On the last: maintainers are exempt from the limit.
EmrServerlessJobRunServiceTest:aRunBeyondMaxConcurrentRunsIsQueuedAndStartsTheMomentARunEnds: with a limit of 1 the second run isQUEUED, listed underQUEUED, and starts when the first ends, with itsqueuedDurationMilliseconds;aRunStillQueuedAtTheQueueTimeoutIsCancelled:CANCELLEDatcreatedAtplus the timeout, never started;aCancelOrAHigherLimitOnTheStartedApplicationStartsTheQueuedRunsInOrderAtOnce: a queued run is cancelled at once, a cancelled running run's slot goes to the next queued one,StopApplicationis refused while runs wait, and a raised limit starts the rest;runsQueueByDefaultFromRelease730AndNeverBeforeRelease700: 7.3.0 andemr-spark-8.0.0queue the 16th run with no configuration, 7.2.0 does not, and 6.15.0 does not even with one;aQueuedContainerRunStartsItsDriverOnlyOnceTheRunningOneHasEnded: in thedockermode the queued run's driver is launched only after the first run ends, and a queued run has no driver to stop;aQueuedBatchRunThatStartsWithoutADurationEndsAsItStarts: with the default duration, a batch run leaving the queue isSUCCESSat once;manyRunsEndingTogetherHandTheirSlotsOnInOrder: 20 runs on five slots replayed in one read, each queued run starting as the run five places ahead ends;anEmptySchedulerConfigurationTurnsQueuingOffOnAReleaseThatQueuesByDefault;aQueuedRunWhoseDriverIsNotItsApplicationsFailsAsItLeavesTheQueueAndTakesNoSlot: ahiverun queued on a Spark application fails as it leaves the queue, and the valid run behind it starts.EmrServerlessServiceTest:aSchedulerConfigurationFillsWhatItLeavesOutAndAnEmptyOneRemovesIt: missing members filled, an empty one removed, out-of-range and non-integer values refused;aStartedApplicationTakesANewMaxConcurrentRunsButNoOtherSchedulerChange;anUpdateRefusedForItsSchedulerConfigurationChangesNothingElse: a refused scheduler value leaves the capacity sent beside it unapplied.Settling an application replays its runs from a priority queue of endings and a deque of queued runs, reads that application's runs alone through an index of run ids by application, and stores what it changed in one write.
EmrServerlessJobRunIntegrationTest.aRunBeyondMaxConcurrentRunsWaitsQueuedUntilTheRunningOneIsCancelled: over the wire, the filled configuration, aQUEUEDstreaming run listed by state, started once the running one is cancelled, and an empty configuration removed. Every EMR Serverless test class but the Docker and nightly ones passes as a whole. Checkstyle,docs-checkandpartition-checkare clean.