Skip to content

Run background jobs in a dedicated worker container - #40

Merged
vitobotta merged 11 commits into
vitobotta:mainfrom
phantomic12:deployment-fixes
Nov 15, 2026
Merged

vitobotta merged 11 commits into
vitobotta:mainfrom
phantomic12:deployment-fixes

Conversation

@phantomic12

Copy link
Copy Markdown
Contributor

Summary

Splits job execution out of the web container and fixes three deployment settings that were silently not doing what the compose file claimed.

  • New worker service running bundle exec bin/jobs. The image's default command is the web server, so the override is required.
  • config/puma.rb: the gate was plugin :solid_queue if ENV["SOLID_QUEUE_IN_PUMA"], and every string is truthy in Ruby. SOLID_QUEUE_IN_PUMA: "false" therefore started the in-Puma supervisor. It now compares the value, so 0/false/no/off disable it.
  • config/environments/production.rb: force_ssl/assume_ssl were hardcoded true, so the FORCE_SSL/ASSUME_SSL values compose passed were ignored, and a plain-HTTP deployment redirected to an https:// URL nothing was listening on.
  • .dockerignore: /data is a root-owned Postgres bind mount, so every docker compose up --build after first boot failed with open .../data: permission denied.
  • .env.example: documents the TLS pair; both still default to true, so existing proxied deployments keep their current posture.

Test plan

Deployed this branch on a clean Ubuntu 24.04 host with Docker 29.1.3 / Compose 2.40.3:

  • docker compose up -d --build brought up db, web, worker; docker compose config validates.
  • Rebuild with ./data already present now succeeds (previously a hard failure).
  • With SOLID_QUEUE_IN_PUMA=false, the web container registers no Solid Queue processes; only the worker's hostname holds Dispatcher/Worker/Scheduler.
  • Enqueued RefreshRecommendationsJob with the worker stopped: it sat ready=1, finished_at=nil. Starting the worker consumed it (Performed ... in 57.79ms, ready=0).
  • FORCE_SSL=false in .env: / redirects to http://…/users/sign_in with no HSTS header, and sign-in succeeds (303 → authenticated / 200).

The compose stack ran Solid Queue inside Puma, but the puma plugin gate tested
the environment variable for presence, so SOLID_QUEUE_IN_PUMA=false started the
supervisor anyway. Compare the value, add a worker service that owns job
execution, honour the FORCE_SSL/ASSUME_SSL values compose already forwarded, and
ignore the root-owned Postgres bind mount that broke every rebuild after first
boot.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
@phantomic12

Copy link
Copy Markdown
Contributor Author

@vitobotta ready for a first review: runs background jobs in a dedicated worker container. Heads up that it is 24 commits behind main — say the word and I will merge upstream in before you look at it.

@vitobotta vitobotta left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First review at af10eac, merged with current main in a throwaway branch. The merge is clean, docker compose config validates, the merged tree runs 494 examples with 0 failures, and RuboCop is clean on the two Ruby config files.

The Puma and TLS gates behave as described: falsey strings disable both, and the compose web service passes false for the Solid Queue plugin. I also checked bin/jobs, the worker command, config/queue.yml, and the recurring entries.

One blocking problem: the recurring HLS cleanup moved to the worker with the rest of the scheduler, but it works on the web container's filesystem and process table. Details on the line below. Two smaller items are on the lines as well: the JOB_CONCURRENCY comment names the wrong dimension, and the Puma gate now enables the supervisor when the variable is unset. The README still describes a single web service.

I did not build the image or repeat the enqueue and consume round trip here; the test plan covers that, and the compose and queue wiring check out.

Comment thread docker-compose.yml Resolved
Comment thread docker-compose.yml Resolved
Comment thread config/puma.rb Resolved
Comment thread .env.example Resolved
The cleanup stops PIDs and deletes tmp/hls paths that only exist in the web container, so it now runs from a web-side loop instead of the worker's scheduler; JOB_CONCURRENCY becomes JOB_PROCESSES to match what it configures, and the Puma supervisor stays opt-in.
@phantomic12

Copy link
Copy Markdown
Contributor Author

@vitobotta all four addressed, and the branch now merges current upstream/main so the head you reviewed with a throwaway branch is what is pushed.

  • Blocking — HLS cleanup in the worker: you are right that it cannot work there: the rows carry PIDs and tmp/hls paths that only exist in the web container. I took the "keep it in the web container" option: the recurring entry is gone from config/recurring.yml, and config/initializers/hls_session_cleanup.rb runs the same HlsSessionManager.cleanup_expired on a 5 minute loop inside the web process (skipped unless Rails::Server, so bin/jobs and consoles never start it, and skipped when SOLID_QUEUE_IN_PUMA owns the scheduler). Shared volumes plus a non-PID stop signal would be the alternative, but it changes the session lifecycle and this keeps the behaviour that was already proven.
  • JOB_CONCURRENCY: renamed to JOB_PROCESSES, which is what config/queue.yml actually configures; the comment now says each process runs three threads.
  • Puma gate: back to opt-in — plugin :solid_queue only when SOLID_QUEUE_IN_PUMA is present and not a falsey spelling, so unset no longer enables the supervisor.
  • Docs: the README table lists FORCE_SSL, ASSUME_SSL, SOLID_QUEUE_IN_PUMA and JOB_PROCESSES, the troubleshooting table has a "jobs never run / recommendations stale" row pointing at the worker, the logs snippet tails web worker, and .env.example documents both job variables.

Checks: zeitwerk:check clean, RuboCop clean on config/puma.rb and the new initializer, config/queue.yml parses, and ruby -c on both Ruby files. I have not built the image or repeated the enqueue/consume round trip, so that part of your test plan is still unverified here.

@phantomic12

Copy link
Copy Markdown
Contributor Author

Ran the missing part of the test plan on a clean clone of the branch: built the image with docker compose build, brought the three-service stack up, and exercised the queue end to end.

  • Build: docker compose build succeeds from a fresh clone (linux/amd64), so the Dockerfile and lockfile are consistent; docker compose config validates.
  • Startup: db healthy, web and worker running, GET /up returns 200.
  • Enqueue/consume: from the web container, RefreshRecommendationsJob.perform_later(0) left SolidQueue::ReadyExecution.count == 1; six seconds later the job is finished=true with ready=0 and claimed=0, and the worker log shows Performing then Performed ... in 26.64ms. Delivery is proven; the job short-circuits on a missing user, so the recommendation logic itself is not exercised here.
  • Worker owns the scheduler: the worker logs Started Supervisor(fork), Dispatcher, two workers, and Scheduler ... recurring_schedule: ["clear_solid_queue_finished_jobs"]. HLS cleanup is not in that schedule, and docker compose logs worker | grep -c HlsSessionCleanup is 0.
  • Web does not run jobs: SOLID_QUEUE_IN_PUMA=false from compose is honoured — there is no bin/jobs process in the web container. The web log does show [HlsSessionCleanup] starting web-side HLS cleanup loop, and HlsSessionManager.cleanup_expired runs there and returns normally.
  • JOB_PROCESSES: recreating the worker with JOB_PROCESSES=2 starts two worker processes, confirming it maps to processes.

The test stack and its images are torn down; the existing deployment on this host was not touched.

@vitobotta vitobotta left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 9ca0960. The four findings from the first pass are addressed:

  • HLS cleanup no longer runs from the worker; the new initializer owns it in the web process.
  • JOB_CONCURRENCY is now JOB_PROCESSES, wired through config/queue.yml, the compose worker, .env.example and the README.
  • The Puma plugin is opt-in again: unset and falsey spellings leave it off, true starts it.
  • The README now documents FORCE_SSL, ASSUME_SSL, SOLID_QUEUE_IN_PUMA and JOB_PROCESSES, tails both services and has a worker troubleshooting row.

Two blocking gaps remain in the new web cleanup, both on the lines below: it is disabled in SOLID_QUEUE_IN_PUMA=true deployments, where the removed recurring task no longer covers it, and the Rails::Server guard misses direct Puma boots (the Procfile). The PR is also CONFLICTING with main in config/environments/production.rb, so it needs a merge or rebase.

Gates at the head: full suite 494/0, RuboCop clean on the two touched Ruby files, docker compose config valid, and both files pass ruby -c.

Minor, not blocking: HLS_CLEANUP_IN_WEB is not documented, and with WEB_CONCURRENCY above 1 every Puma worker starts its own cleanup loop.

The loop was skipped exactly when the Puma scheduler owned jobs, and Rails::Server is never set by the Procfile's direct Puma boot; it now starts whenever Puma is loaded, takes a per-container lock so clustered workers cannot double-run it, and is documented.
# Conflicts:
#	config/environments/production.rb
defined?(Puma) is true in any Rails boot because the gem sits in the default bundle, so the worker would have started a loop too; the hook runs only in the process that evaluates config/puma.rb, and a re-entry guard keeps phased restarts single.
Puma evaluates config/puma.rb before Rails boots, so #present? raised NoMethodError and the server could not start at all.
The initializer kept its own after_initialize call, so bin/jobs started a loop as well; the web server is the only process that should own it.
@phantomic12

phantomic12 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor Author

@vitobotta both blocking items are fixed and verified in a built container, head 28a6fe7.

  • Both job modes: the SOLID_QUEUE_IN_PUMA gate is gone. The loop is started from config/puma.rb (after_booted { HlsCleanup.start }), so the web server runs it whether the scheduler lives in Puma or in the worker.
  • Procfile boots: the initializer no longer looks at Rails::Server. defined?(Puma) turned out to be useless too — the gem is in the default bundle, so it is defined in a plain bin/jobs boot as well — which is why the trigger lives in the Puma config: only the process that reads that file can start it.
  • WEB_CONCURRENCY and restarts: the loop takes a per-container flock around each run and releases it afterwards, so clustered Puma workers never run it at once and a replacement process takes over on the next cycle. HLS_CLEANUP_IN_WEB is documented in .env.example and the README table.

Container evidence, from a fresh clone with the image built by docker compose build:

Boot Trigger HLS loop Result
bundle exec puma -t 5:5 -p 5000 with SOLID_QUEUE_IN_PUMA=true (Procfile) direct Puma started Solid Queue supervisor + scheduler start inside Puma
image default command (thruster + rails server) Puma config started Server started, /up 200
bundle exec bin/jobs (worker) none not started (grep -c = 0) supervisor, worker, scheduler only, recurring schedule is ["clear_solid_queue_finished_jobs"]

The full three-service stack also still works on this head: GET /up 200, a job enqueued from web went ready=1 → finished=true, ready=0 after the worker picked it up, and the web log has exactly one [HlsSessionCleanup] starting line while the worker log has none.

Two real bugs turned up while doing this, both fixed here: the previous plugin :solid_queue guard called present? in config/puma.rb, where ActiveSupport is not loaded, so every Puma boot raised NoMethodError — the container would not start at all; and the initializer's own after_initialize call was still in place, which is what made the worker start a loop in the first boot test.

@vitobotta vitobotta left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both previous blocking findings are fixed. The initializer no longer starts a loop, and config/puma.rb now starts HlsCleanup from Puma 8's after_booted hook. That covers direct Puma, bin/rails server, and both job layouts, while bin/jobs never loads the Puma config.

One new blocking issue: .env.example now has two TLS sections. The later section repeats FORCE_SSL and ASSUME_SSL with true, and Docker Compose uses the last value. A quickstart copy therefore resolves both to true, so plain HTTP on localhost still redirects and the documented quickstart fails. I confirmed this with docker compose config on a copy of the example file: the web service receives FORCE_SSL=true and ASSUME_SSL=true.

Keep one pair of these variables. If the quickstart is plain HTTP, set both to false there and document the TLS-terminating case separately.

Gates after a test merge with current main: 554 RSpec examples, 0 failures; 578 JavaScript tests, 0 failures; RuboCop 186 files, no offenses; Brakeman clean. docker compose config is valid, but it resolves the duplicated TLS values as described above.

Comment thread .env.example Resolved
The example file grew a second FORCE_SSL/ASSUME_SSL pair when the TLS block
moved up to the app configuration section; Compose takes the last value, so
the quickstart's false flags were overwritten by true and plain HTTP on
localhost redirected to https.
@phantomic12

Copy link
Copy Markdown
Contributor Author

The duplicate TLS section is gone at fb30835. .env.example now defines FORCE_SSL and ASSUME_SSL exactly once, in the app-configuration block, as false for the quickstart, and the trailing note there keeps the TLS-terminating case documented ("set both back to true, or delete the lines" — Compose defaults both to true, as the README table says).

Verified with a copy of the example as .env: docker compose config now resolves FORCE_SSL="false" and ASSUME_SSL="false" for every service, so the quickstart stays on plain HTTP. I checked the file kept its CRLF endings and that no other key was touched (7 lines deleted, nothing added).

One thing to flag that is not from this change: scan_ruby now fails on this head because the locked puma 8.0.2 has newly published advisories (GHSA-m5xw-3hrw-qcmf and five others, fix in 8.0.3). Main is red for the same reason — the run on 95d4d91 fails scan_ruby with the identical list — so it needs a Puma bump on main rather than a fix in this PR. lint and scan_js pass here.

@vitobotta vitobotta left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The duplicate TLS pair is gone at fb30835. .env.example now defines FORCE_SSL and ASSUME_SSL once, and docker compose config on a copy resolves both to false for the web and worker services. The seven earlier threads are resolved.

A test merge with current main is clean. Gates: 554 RSpec examples, 0 failures; 578 JavaScript tests, 0 failures; RuboCop inspected 186 files with no offenses; Brakeman reports no warnings.

The only failing check is scan_ruby, which reports Puma 8.0.2 advisories and asks for 8.0.3 or later. The same check fails on main at 95d4d91, so this PR does not introduce it. Bump Puma separately from this deployment change.

Approved.

@vitobotta
vitobotta merged commit 029ca3f into vitobotta:main Nov 15, 2026
2 of 3 checks passed
@vitobotta

Copy link
Copy Markdown
Owner

Merged. Thanks @phantomic12 for the dedicated worker container and the HLS cleanup fixes. A new release will be made soon.

@phantomic12
phantomic12 deleted the deployment-fixes branch November 15, 2026 22:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants