Repository navigation
Run background jobs in a dedicated worker container - #40
Conversation
The compose stack ran Solid Queue inside Puma, but the puma plugin gate tested the environment variable for presence, so SOLID_QUEUE_IN_PUMA=false started the supervisor anyway. Compare the value, add a worker service that owns job execution, honour the FORCE_SSL/ASSUME_SSL values compose already forwarded, and ignore the root-owned Postgres bind mount that broke every rebuild after first boot. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
|
@vitobotta ready for a first review: runs background jobs in a dedicated worker container. Heads up that it is 24 commits behind |
vitobotta
left a comment
There was a problem hiding this comment.
First review at af10eac, merged with current main in a throwaway branch. The merge is clean, docker compose config validates, the merged tree runs 494 examples with 0 failures, and RuboCop is clean on the two Ruby config files.
The Puma and TLS gates behave as described: falsey strings disable both, and the compose web service passes false for the Solid Queue plugin. I also checked bin/jobs, the worker command, config/queue.yml, and the recurring entries.
One blocking problem: the recurring HLS cleanup moved to the worker with the rest of the scheduler, but it works on the web container's filesystem and process table. Details on the line below. Two smaller items are on the lines as well: the JOB_CONCURRENCY comment names the wrong dimension, and the Puma gate now enables the supervisor when the variable is unset. The README still describes a single web service.
I did not build the image or repeat the enqueue and consume round trip here; the test plan covers that, and the compose and queue wiring check out.
The cleanup stops PIDs and deletes tmp/hls paths that only exist in the web container, so it now runs from a web-side loop instead of the worker's scheduler; JOB_CONCURRENCY becomes JOB_PROCESSES to match what it configures, and the Puma supervisor stays opt-in.
# Conflicts: # README.md
|
@vitobotta all four addressed, and the branch now merges current
Checks: |
|
Ran the missing part of the test plan on a clean clone of the branch: built the image with
The test stack and its images are torn down; the existing deployment on this host was not touched. |
vitobotta
left a comment
There was a problem hiding this comment.
Re-reviewed at 9ca0960. The four findings from the first pass are addressed:
- HLS cleanup no longer runs from the worker; the new initializer owns it in the web process.
JOB_CONCURRENCYis nowJOB_PROCESSES, wired throughconfig/queue.yml, the compose worker,.env.exampleand the README.- The Puma plugin is opt-in again: unset and falsey spellings leave it off, true starts it.
- The README now documents
FORCE_SSL,ASSUME_SSL,SOLID_QUEUE_IN_PUMAandJOB_PROCESSES, tails both services and has a worker troubleshooting row.
Two blocking gaps remain in the new web cleanup, both on the lines below: it is disabled in SOLID_QUEUE_IN_PUMA=true deployments, where the removed recurring task no longer covers it, and the Rails::Server guard misses direct Puma boots (the Procfile). The PR is also CONFLICTING with main in config/environments/production.rb, so it needs a merge or rebase.
Gates at the head: full suite 494/0, RuboCop clean on the two touched Ruby files, docker compose config valid, and both files pass ruby -c.
Minor, not blocking: HLS_CLEANUP_IN_WEB is not documented, and with WEB_CONCURRENCY above 1 every Puma worker starts its own cleanup loop.
The loop was skipped exactly when the Puma scheduler owned jobs, and Rails::Server is never set by the Procfile's direct Puma boot; it now starts whenever Puma is loaded, takes a per-container lock so clustered workers cannot double-run it, and is documented.
# Conflicts: # config/environments/production.rb
defined?(Puma) is true in any Rails boot because the gem sits in the default bundle, so the worker would have started a loop too; the hook runs only in the process that evaluates config/puma.rb, and a re-entry guard keeps phased restarts single.
Puma evaluates config/puma.rb before Rails boots, so #present? raised NoMethodError and the server could not start at all.
The initializer kept its own after_initialize call, so bin/jobs started a loop as well; the web server is the only process that should own it.
|
@vitobotta both blocking items are fixed and verified in a built container, head
Container evidence, from a fresh clone with the image built by
The full three-service stack also still works on this head: Two real bugs turned up while doing this, both fixed here: the previous |
vitobotta
left a comment
There was a problem hiding this comment.
Both previous blocking findings are fixed. The initializer no longer starts a loop, and config/puma.rb now starts HlsCleanup from Puma 8's after_booted hook. That covers direct Puma, bin/rails server, and both job layouts, while bin/jobs never loads the Puma config.
One new blocking issue: .env.example now has two TLS sections. The later section repeats FORCE_SSL and ASSUME_SSL with true, and Docker Compose uses the last value. A quickstart copy therefore resolves both to true, so plain HTTP on localhost still redirects and the documented quickstart fails. I confirmed this with docker compose config on a copy of the example file: the web service receives FORCE_SSL=true and ASSUME_SSL=true.
Keep one pair of these variables. If the quickstart is plain HTTP, set both to false there and document the TLS-terminating case separately.
Gates after a test merge with current main: 554 RSpec examples, 0 failures; 578 JavaScript tests, 0 failures; RuboCop 186 files, no offenses; Brakeman clean. docker compose config is valid, but it resolves the duplicated TLS values as described above.
The example file grew a second FORCE_SSL/ASSUME_SSL pair when the TLS block moved up to the app configuration section; Compose takes the last value, so the quickstart's false flags were overwritten by true and plain HTTP on localhost redirected to https.
|
The duplicate TLS section is gone at Verified with a copy of the example as One thing to flag that is not from this change: |
vitobotta
left a comment
There was a problem hiding this comment.
The duplicate TLS pair is gone at fb30835. .env.example now defines FORCE_SSL and ASSUME_SSL once, and docker compose config on a copy resolves both to false for the web and worker services. The seven earlier threads are resolved.
A test merge with current main is clean. Gates: 554 RSpec examples, 0 failures; 578 JavaScript tests, 0 failures; RuboCop inspected 186 files with no offenses; Brakeman reports no warnings.
The only failing check is scan_ruby, which reports Puma 8.0.2 advisories and asks for 8.0.3 or later. The same check fails on main at 95d4d91, so this PR does not introduce it. Bump Puma separately from this deployment change.
Approved.
|
Merged. Thanks @phantomic12 for the dedicated worker container and the HLS cleanup fixes. A new release will be made soon. |
Summary
Splits job execution out of the web container and fixes three deployment settings that were silently not doing what the compose file claimed.
workerservice runningbundle exec bin/jobs. The image's default command is the web server, so the override is required.config/puma.rb: the gate wasplugin :solid_queue if ENV["SOLID_QUEUE_IN_PUMA"], and every string is truthy in Ruby.SOLID_QUEUE_IN_PUMA: "false"therefore started the in-Puma supervisor. It now compares the value, so0/false/no/offdisable it.config/environments/production.rb:force_ssl/assume_sslwere hardcodedtrue, so theFORCE_SSL/ASSUME_SSLvalues compose passed were ignored, and a plain-HTTP deployment redirected to anhttps://URL nothing was listening on..dockerignore:/datais a root-owned Postgres bind mount, so everydocker compose up --buildafter first boot failed withopen .../data: permission denied..env.example: documents the TLS pair; both still default totrue, so existing proxied deployments keep their current posture.Test plan
Deployed this branch on a clean Ubuntu 24.04 host with Docker 29.1.3 / Compose 2.40.3:
docker compose up -d --buildbrought updb,web,worker;docker compose configvalidates../dataalready present now succeeds (previously a hard failure).SOLID_QUEUE_IN_PUMA=false, the web container registers no Solid Queue processes; only the worker's hostname holdsDispatcher/Worker/Scheduler.RefreshRecommendationsJobwith the worker stopped: it satready=1, finished_at=nil. Starting the worker consumed it (Performed ... in 57.79ms,ready=0).FORCE_SSL=falsein.env:/redirects tohttp://…/users/sign_inwith no HSTS header, and sign-in succeeds (303 → authenticated/200).