Skip to content

docs: disambiguate the daisy/daisy collisions the rename left in dev/, refresh the REFACTOR roadmap, merge the split CHANGELOG headings #77

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pattonw merged 2 commits into from
Aug 4, 2026
Merged

docs: disambiguate the daisy/daisy collisions the rename left in dev/, refresh the REFACTOR roadmap, merge the split CHANGELOG headings #77

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6e35d28
Disambiguate the daisy/daisy collisions the rename left in dev/
Aug 4, 2026
055fdfb
Merge branch 'v2.0' into docs/repair-rename-collisions
pattonw Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
271 changes: 127 additions & 144 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,49 +7,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Removed
### Added

- **`Task(worker_processes=...)` is gone, and with it in-process (thread)
workers.** Every distributed worker now runs in its own OS process, as in
daisy 1.x. There is nothing to configure: whatever a task's
`process_function` is, the runner wraps it into a spawn function that
launches `python -m daisy._subprocess_worker`, which either drives the
`Client.acquire_block()` loop (1-arg block function) or calls the function
once and lets it drive its own loop (0-arg worker function, which may in
turn `srun`/`sbatch` a further process).
- `failures`: per-block count of failed attempts, written whenever tracking
is on at all (independent of `resource_tracking`), covering both
worker-reported failures and timeout reclaims.

Why: 0-arg worker functions had no choice — they always ran on a server
thread — so a worker that did its work inline instead of shelling out got
no parallelism at all. Measured on 8 CPU-bound blocks, 1 worker vs 4:
**0.59×**, i.e. slower than a single worker. The same workload in worker
processes measures **3.00×**. volara's default worker has exactly this
shape. Keeping thread execution to serve the one case where it won (block
functions that release the GIL for essentially their whole runtime, by
~15%) meant two execution models, two measurement paths, and a `timeout=`
that could not preempt.
- Lint/type CI (`lint.yaml`): `ruff check` + `ruff format --check` with a
pinned rule set (pyflakes, pycodestyle errors, import sorting) and
`ty check` over the python package, both via `uv run --only-group lint`
(no Rust build needed in CI). The repo is fully clean under all three;
the self-comparing upstream `benchmarks/` are excluded from lint pending
a rewrite.

**Breaking changes for callers:**
- Wheel-building CI (`publish.yaml`): manylinux/musllinux x86_64 + aarch64,
macOS x86_64 + arm64, Windows x64, plus sdist; a built wheel is
smoke-tested (install without a Rust toolchain, run a tiny blockwise
task) before any publish; publishes to PyPI on version tags.

- `Task(worker_processes=...)` raises `TypeError`. Delete the argument.
Code that passed `False` to observe in-process state should use
`run_blockwise(..., multiprocessing=False)`, which runs the original
function in-process, single-threaded.
- **0-arg worker functions must now be picklable**, because they are
shipped to a subprocess rather than called in place. v2 cannot fork —
the server runs a tokio runtime per worker thread, and forking a
multithreaded process is unsafe — so it spawns and serializes. An
unpicklable function fails before the run starts with guidance.
- **No cheap shared read-only memory.** Workers can no longer close over a
large array. `numpy.memmap` a file instead.
- **`Task(timeout=...)` now always preempts.** There is no longer a mode in
which a timed-out block keeps running and can double-apply its effects
concurrently with the retry.
- Spawn functions may declare a keyword-only `context` parameter
(`def start_worker(*, context):`) to receive their worker's
`daisy.Context` as an argument — a race-free alternative to reading the
process-global `DAISY_CONTEXT` environment variable, which concurrent
slow spawn functions can observe with a later worker's value. The env
var keeps being set for 0-arg spawn functions and worker children.
`Context.from_env_string(...)` parses an encoded context without
touching the environment. The built-in subprocess workers now set the
child's `DAISY_CONTEXT` deterministically from the argument.

Also: worker identity is race-free for both spawn signatures. Each child
gets its own environment, so the `DAISY_CONTEXT` race that the keyword-only
`context` parameter was introduced to dodge cannot happen; the warning
about it is gone. And `daisy.logging` settings are carried in the worker
payload, since a spawned child inherits no process globals.
- Resumed runs now emit an INFO record per task on the `daisy._progress`
logger stating how many blocks were skipped via done markers and how to
reprocess them.

- Optional dependency extra `daisy[worker-processes]` installing
`cloudpickle` for lambda/closure support in worker processes.

### Changed

Expand All @@ -62,8 +53,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
cluster workers. An absent key (hand-built `Context`, older server)
means off.

### Changed

- **Run statistics are now an optional per-block layer.** Set
`Task(resource_tracking=True)` and every block comes back carrying what it
cost — wall time, CPU time, peak RSS, IO bytes — measured inside whoever
Expand Down Expand Up @@ -111,15 +100,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`set_done_marker_basedir` / `get_done_marker_basedir` keep working and
emit a `DeprecationWarning`.

### Added

- `failures`: per-block count of failed attempts, written whenever tracking
is on at all (independent of `resource_tracking`), covering both
worker-reported failures and timeout reclaims.


### Changed

- Subprocess-worker payloads are serialized with **cloudpickle** instead of
dill (optional dependency `daisy[worker-processes]` now installs
cloudpickle). Both ship functions by value; they differ on *modules* a
Expand All @@ -142,8 +122,98 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
such state into the block function instead. Both changes make local
subprocess runs behave like cluster runs.

### Added
- **Blocks always have a timeout.** `Task(timeout=...)` defaults to 600
seconds and can no longer be disabled (`None` = the default; values
<= 0 raise ValueError). A hung block can therefore wedge a run — or
its shutdown — for at most the block deadline: the subprocess worker
self-kills at the deadline, the block is reclaimed and retried, and
shutdown joins complete. Failure surfaces attribute timeout reclaims
(`TaskState.timeout_reclaim_count` / `timeout_secs`, plus run-summary
and abandonment-error hints pointing at `Task(timeout=...)`).

- **Done-marker tracking is now opt-in.** `Task(done_marker_path=None)` only
resolves to a marker when `set_done_marker_basedir(...)` has been called;
the fallback to the logging basedir (`./daisy_logs/<task_id>`, relative to
the current working directory) is removed. Previously a rerun of a script
whose code had changed could silently skip every block a prior run had
marked done. Explicit `done_marker_path="..."` and `done_marker_path=False`
behave as before. To restore the old behavior, call
`daisy.set_done_marker_basedir(...)` once at pipeline start.

- **Block functions run in worker subprocesses.** A 1-arg `process_function`
on the distributed run paths is serialized (with `cloudpickle` when
installed; stdlib `pickle` otherwise) and executed by real OS worker
processes launched as `python -m daisy._subprocess_worker`, rather than on
GIL-sharing threads inside the server process. Serial execution
(`run_blockwise(..., multiprocessing=False)`) is unchanged.

Why: across workload mixes (16 workers, 96 × ~100 ms blocks), threads win
only when the block function releases the GIL for essentially its entire
runtime — pure I/O waits or single-threaded C-library calls — and then only
by 13–17%. With just 10% pure-python glue, threads are 1.7× slower; at 30%
python, 8× slower; at 100% python, 28× slower. Worker processes are flat
across every mix. (Thread execution was briefly retained as an opt-out via
`worker_processes=False`; see the Removed section below for why it went.)

**Resource implication**: `max_workers=N` means N python interpreter
processes (each importing your function's modules) rather than N threads in
one process. Budget memory accordingly.

`Task(timeout=...)` gains true preemption: a block exceeding the deadline
kills its worker process (visible as a dirty exit, bounded by
`max_worker_restarts`) instead of leaving a runaway thread behind.

### Removed

- **`Task(worker_processes=...)` is gone, and with it in-process (thread)
workers.** Every distributed worker now runs in its own OS process, as in
daisy 1.x. There is nothing to configure: whatever a task's
`process_function` is, the runner wraps it into a spawn function that
launches `python -m daisy._subprocess_worker`, which either drives the
`Client.acquire_block()` loop (1-arg block function) or calls the function
once and lets it drive its own loop (0-arg worker function, which may in
turn `srun`/`sbatch` a further process).

Why: 0-arg worker functions had no choice — they always ran on a server
thread — so a worker that did its work inline instead of shelling out got
no parallelism at all. Measured on 8 CPU-bound blocks, 1 worker vs 4:
**0.59×**, i.e. slower than a single worker. The same workload in worker
processes measures **3.00×**. volara's default worker has exactly this
shape. Keeping thread execution to serve the one case where it won (block
functions that release the GIL for essentially their whole runtime, by
~15%) meant two execution models, two measurement paths, and a `timeout=`
that could not preempt.

**Breaking changes for callers:**

- `Task(worker_processes=...)` raises `TypeError`. Delete the argument.
Code that passed `False` to observe in-process state should use
`run_blockwise(..., multiprocessing=False)`, which runs the original
function in-process, single-threaded.
- **0-arg worker functions must now be picklable**, because they are
shipped to a subprocess rather than called in place. v2 cannot fork —
the server runs a tokio runtime per worker thread, and forking a
multithreaded process is unsafe — so it spawns and serializes. An
unpicklable function fails before the run starts with guidance.
- **No cheap shared read-only memory.** Workers can no longer close over a
large array. `numpy.memmap` a file instead.
- **`Task(timeout=...)` now always preempts.** There is no longer a mode in
which a timed-out block keeps running and can double-apply its effects
concurrently with the retry.

Also: worker identity is race-free for both spawn signatures. Each child
gets its own environment, so the `DAISY_CONTEXT` race that the keyword-only
`context` parameter was introduced to dodge cannot happen; the warning
about it is gone. And `daisy.logging` settings are carried in the worker
payload, since a spawned child inherits no process globals.

- The `funlib.persistence.Array` monkey-patching in the v1-compat layer.
daisy and funlib.persistence are unrelated packages that merely share
`funlib.geometry`; daisy no longer imports or modifies persistence.
Compat-surface blocks already carry `funlib.geometry` ROIs (the
`_BlockProxy` boundary), so v1-style code can index persistence Arrays
with them directly — no patching required. Native `daisy.v2` ROIs are
not accepted by persistence; convert explicitly if you mix surfaces.
- Lint/type CI (`lint.yaml`): `ruff check` + `ruff format --check` with a
pinned rule set (pyflakes, pycodestyle errors, import sorting) and
`ty check` over the python package, both via `uv run --only-group lint`
Expand Down Expand Up @@ -241,113 +311,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
and custom workers); thread mode continues to determine success by
exception only, as it always has in v2.

### Documentation

- Documented the blocking-spawn contract: spawn functions must run for the
worker's lifetime (`sbatch --wait` / `bsub -K` / `subprocess.run`);
submit-and-return spawns over-submit by up to `max_worker_restarts` jobs
and can abandon a task whose jobs are still queued. Connection-aware
worker accounting is planned.

### Changed

- **Blocks always have a timeout.** `Task(timeout=...)` defaults to 600
seconds and can no longer be disabled (`None` = the default; values
<= 0 raise ValueError). A hung block can therefore wedge a run — or
its shutdown — for at most the block deadline: the subprocess worker
self-kills at the deadline, the block is reclaimed and retried, and
shutdown joins complete. Failure surfaces attribute timeout reclaims
(`TaskState.timeout_reclaim_count` / `timeout_secs`, plus run-summary
and abandonment-error hints pointing at `Task(timeout=...)`).

### Fixed

- Worker-log stream proxies are now uninstalled at the end of each run;
previously the first run's `sys.stdout`/`sys.stderr` were captured
forever and every later run's execution summary was written to the
(possibly closed or replaced) original streams.

### Removed

- The `funlib.persistence.Array` monkey-patching in the v1-compat layer.
daisy and funlib.persistence are unrelated packages that merely share
`funlib.geometry`; daisy no longer imports or modifies persistence.
Compat-surface blocks already carry `funlib.geometry` ROIs (the
`_BlockProxy` boundary), so v1-style code can index persistence Arrays
with them directly — no patching required. Native `daisy.v2` ROIs are
not accepted by persistence; convert explicitly if you mix surfaces.

### Added

- Wheel-building CI (`publish.yaml`): manylinux/musllinux x86_64 + aarch64,
macOS x86_64 + arm64, Windows x64, plus sdist; a built wheel is
smoke-tested (install without a Rust toolchain, run a tiny blockwise
task) before any publish; publishes to PyPI on version tags.

- Spawn functions may declare a keyword-only `context` parameter
(`def start_worker(*, context):`) to receive their worker's
`daisy.Context` as an argument — a race-free alternative to reading the
process-global `DAISY_CONTEXT` environment variable, which concurrent
slow spawn functions can observe with a later worker's value. The env
var keeps being set for 0-arg spawn functions and worker children.
`Context.from_env_string(...)` parses an encoded context without
touching the environment. The built-in subprocess workers now set the
child's `DAISY_CONTEXT` deterministically from the argument.

### Changed

- **Done-marker tracking is now opt-in.** `Task(done_marker_path=None)` only
resolves to a marker when `set_done_marker_basedir(...)` has been called;
the fallback to the logging basedir (`./daisy_logs/<task_id>`, relative to
the current working directory) is removed. Previously a rerun of a script
whose code had changed could silently skip every block a prior run had
marked done. Explicit `done_marker_path="..."` and `done_marker_path=False`
behave as before. To restore the old behavior, call
`daisy.set_done_marker_basedir(...)` once at pipeline start.

### Added

- Resumed runs now emit an INFO record per task on the `daisy._progress`
logger stating how many blocks were skipped via done markers and how to
reprocess them.
- Worker starts are now bounded by a hard per-task budget of `max_workers + max_worker_restarts`, regardless of how or why previous workers exited. Previously only dirty exits counted toward the restart cap, so a worker that exited cleanly without processing blocks (e.g. `subprocess.run(..., check=False)` around a command that fails to start) respawned forever and the run never terminated. Workers are expected to be long-running; the recycle-after-N-blocks pattern is not supported — size `max_worker_restarts` for expected worker deaths (preemption, walltime), or resume via done markers. See `docs/source/design/ABANDONMENT.md`.

### Documentation

- Documented the blocking-spawn contract: spawn functions must run for the
worker's lifetime (`sbatch --wait` / `bsub -K` / `subprocess.run`);
submit-and-return spawns over-submit by up to `max_worker_restarts` jobs
and can abandon a task whose jobs are still queued. Connection-aware
worker accounting is planned.

- Done markers: documented the layout-hash limitation for grown volumes
(extending `total_roi` currently invalidates the whole marker) and the
planned in-place migration enhancement, with interim workarounds.
(adversarial suite case f05)

- Worker starts are now bounded by a hard per-task budget of `max_workers + max_worker_restarts`, regardless of how or why previous workers exited. Previously only dirty exits counted toward the restart cap, so a worker that exited cleanly without processing blocks (e.g. `subprocess.run(..., check=False)` around a command that fails to start) respawned forever and the run never terminated. Workers are expected to be long-running; the recycle-after-N-blocks pattern is not supported — size `max_worker_restarts` for expected worker deaths (preemption, walltime), or resume via done markers. See `docs/source/design/ABANDONMENT.md`.

- **Block functions run in worker subprocesses.** A 1-arg `process_function`
on the distributed run paths is serialized (with `cloudpickle` when
installed; stdlib `pickle` otherwise) and executed by real OS worker
processes launched as `python -m daisy._subprocess_worker`, rather than on
GIL-sharing threads inside the server process. Serial execution
(`run_blockwise(..., multiprocessing=False)`) is unchanged.

Why: across workload mixes (16 workers, 96 × ~100 ms blocks), threads win
only when the block function releases the GIL for essentially its entire
runtime — pure I/O waits or single-threaded C-library calls — and then only
by 13–17%. With just 10% pure-python glue, threads are 1.7× slower; at 30%
python, 8× slower; at 100% python, 28× slower. Worker processes are flat
across every mix. (Thread execution was briefly retained as an opt-out via
`worker_processes=False`; see the Removed section above for why it went.)

**Resource implication**: `max_workers=N` means N python interpreter
processes (each importing your function's modules) rather than N threads in
one process. Budget memory accordingly.

`Task(timeout=...)` gains true preemption: a block exceeding the deadline
kills its worker process (visible as a dirty exit, bounded by
`max_worker_restarts`) instead of leaving a runaway thread behind.

### Added

- Optional dependency extra `daisy[worker-processes]` installing
`cloudpickle` for lambda/closure support in worker processes.

## [2.0.0] — 2026-04-27

### Overview
Expand Down
Loading
Loading