diff --git a/CHANGELOG.md b/CHANGELOG.md index a9241cc6..201f92b1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,49 +7,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -### Removed +### Added -- **`Task(worker_processes=...)` is gone, and with it in-process (thread) - workers.** Every distributed worker now runs in its own OS process, as in - daisy 1.x. There is nothing to configure: whatever a task's - `process_function` is, the runner wraps it into a spawn function that - launches `python -m daisy._subprocess_worker`, which either drives the - `Client.acquire_block()` loop (1-arg block function) or calls the function - once and lets it drive its own loop (0-arg worker function, which may in - turn `srun`/`sbatch` a further process). +- `failures`: per-block count of failed attempts, written whenever tracking + is on at all (independent of `resource_tracking`), covering both + worker-reported failures and timeout reclaims. - Why: 0-arg worker functions had no choice — they always ran on a server - thread — so a worker that did its work inline instead of shelling out got - no parallelism at all. Measured on 8 CPU-bound blocks, 1 worker vs 4: - **0.59×**, i.e. slower than a single worker. The same workload in worker - processes measures **3.00×**. volara's default worker has exactly this - shape. Keeping thread execution to serve the one case where it won (block - functions that release the GIL for essentially their whole runtime, by - ~15%) meant two execution models, two measurement paths, and a `timeout=` - that could not preempt. +- Lint/type CI (`lint.yaml`): `ruff check` + `ruff format --check` with a + pinned rule set (pyflakes, pycodestyle errors, import sorting) and + `ty check` over the python package, both via `uv run --only-group lint` + (no Rust build needed in CI). The repo is fully clean under all three; + the self-comparing upstream `benchmarks/` are excluded from lint pending + a rewrite. - **Breaking changes for callers:** +- Wheel-building CI (`publish.yaml`): manylinux/musllinux x86_64 + aarch64, + macOS x86_64 + arm64, Windows x64, plus sdist; a built wheel is + smoke-tested (install without a Rust toolchain, run a tiny blockwise + task) before any publish; publishes to PyPI on version tags. - - `Task(worker_processes=...)` raises `TypeError`. Delete the argument. - Code that passed `False` to observe in-process state should use - `run_blockwise(..., multiprocessing=False)`, which runs the original - function in-process, single-threaded. - - **0-arg worker functions must now be picklable**, because they are - shipped to a subprocess rather than called in place. v2 cannot fork — - the server runs a tokio runtime per worker thread, and forking a - multithreaded process is unsafe — so it spawns and serializes. An - unpicklable function fails before the run starts with guidance. - - **No cheap shared read-only memory.** Workers can no longer close over a - large array. `numpy.memmap` a file instead. - - **`Task(timeout=...)` now always preempts.** There is no longer a mode in - which a timed-out block keeps running and can double-apply its effects - concurrently with the retry. +- Spawn functions may declare a keyword-only `context` parameter + (`def start_worker(*, context):`) to receive their worker's + `daisy.Context` as an argument — a race-free alternative to reading the + process-global `DAISY_CONTEXT` environment variable, which concurrent + slow spawn functions can observe with a later worker's value. The env + var keeps being set for 0-arg spawn functions and worker children. + `Context.from_env_string(...)` parses an encoded context without + touching the environment. The built-in subprocess workers now set the + child's `DAISY_CONTEXT` deterministically from the argument. - Also: worker identity is race-free for both spawn signatures. Each child - gets its own environment, so the `DAISY_CONTEXT` race that the keyword-only - `context` parameter was introduced to dodge cannot happen; the warning - about it is gone. And `daisy.logging` settings are carried in the worker - payload, since a spawned child inherits no process globals. +- Resumed runs now emit an INFO record per task on the `daisy._progress` + logger stating how many blocks were skipped via done markers and how to + reprocess them. + +- Optional dependency extra `daisy[worker-processes]` installing + `cloudpickle` for lambda/closure support in worker processes. ### Changed @@ -62,8 +53,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 cluster workers. An absent key (hand-built `Context`, older server) means off. -### Changed - - **Run statistics are now an optional per-block layer.** Set `Task(resource_tracking=True)` and every block comes back carrying what it cost — wall time, CPU time, peak RSS, IO bytes — measured inside whoever @@ -111,15 +100,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `set_done_marker_basedir` / `get_done_marker_basedir` keep working and emit a `DeprecationWarning`. -### Added - -- `failures`: per-block count of failed attempts, written whenever tracking - is on at all (independent of `resource_tracking`), covering both - worker-reported failures and timeout reclaims. - - -### Changed - - Subprocess-worker payloads are serialized with **cloudpickle** instead of dill (optional dependency `daisy[worker-processes]` now installs cloudpickle). Both ship functions by value; they differ on *modules* a @@ -142,8 +122,98 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 such state into the block function instead. Both changes make local subprocess runs behave like cluster runs. -### Added +- **Blocks always have a timeout.** `Task(timeout=...)` defaults to 600 + seconds and can no longer be disabled (`None` = the default; values + <= 0 raise ValueError). A hung block can therefore wedge a run — or + its shutdown — for at most the block deadline: the subprocess worker + self-kills at the deadline, the block is reclaimed and retried, and + shutdown joins complete. Failure surfaces attribute timeout reclaims + (`TaskState.timeout_reclaim_count` / `timeout_secs`, plus run-summary + and abandonment-error hints pointing at `Task(timeout=...)`). + +- **Done-marker tracking is now opt-in.** `Task(done_marker_path=None)` only + resolves to a marker when `set_done_marker_basedir(...)` has been called; + the fallback to the logging basedir (`./daisy_logs/`, relative to + the current working directory) is removed. Previously a rerun of a script + whose code had changed could silently skip every block a prior run had + marked done. Explicit `done_marker_path="..."` and `done_marker_path=False` + behave as before. To restore the old behavior, call + `daisy.set_done_marker_basedir(...)` once at pipeline start. + +- **Block functions run in worker subprocesses.** A 1-arg `process_function` + on the distributed run paths is serialized (with `cloudpickle` when + installed; stdlib `pickle` otherwise) and executed by real OS worker + processes launched as `python -m daisy._subprocess_worker`, rather than on + GIL-sharing threads inside the server process. Serial execution + (`run_blockwise(..., multiprocessing=False)`) is unchanged. + + Why: across workload mixes (16 workers, 96 × ~100 ms blocks), threads win + only when the block function releases the GIL for essentially its entire + runtime — pure I/O waits or single-threaded C-library calls — and then only + by 13–17%. With just 10% pure-python glue, threads are 1.7× slower; at 30% + python, 8× slower; at 100% python, 28× slower. Worker processes are flat + across every mix. (Thread execution was briefly retained as an opt-out via + `worker_processes=False`; see the Removed section below for why it went.) + + **Resource implication**: `max_workers=N` means N python interpreter + processes (each importing your function's modules) rather than N threads in + one process. Budget memory accordingly. + + `Task(timeout=...)` gains true preemption: a block exceeding the deadline + kills its worker process (visible as a dirty exit, bounded by + `max_worker_restarts`) instead of leaving a runaway thread behind. + +### Removed + +- **`Task(worker_processes=...)` is gone, and with it in-process (thread) + workers.** Every distributed worker now runs in its own OS process, as in + daisy 1.x. There is nothing to configure: whatever a task's + `process_function` is, the runner wraps it into a spawn function that + launches `python -m daisy._subprocess_worker`, which either drives the + `Client.acquire_block()` loop (1-arg block function) or calls the function + once and lets it drive its own loop (0-arg worker function, which may in + turn `srun`/`sbatch` a further process). + + Why: 0-arg worker functions had no choice — they always ran on a server + thread — so a worker that did its work inline instead of shelling out got + no parallelism at all. Measured on 8 CPU-bound blocks, 1 worker vs 4: + **0.59×**, i.e. slower than a single worker. The same workload in worker + processes measures **3.00×**. volara's default worker has exactly this + shape. Keeping thread execution to serve the one case where it won (block + functions that release the GIL for essentially their whole runtime, by + ~15%) meant two execution models, two measurement paths, and a `timeout=` + that could not preempt. + + **Breaking changes for callers:** + + - `Task(worker_processes=...)` raises `TypeError`. Delete the argument. + Code that passed `False` to observe in-process state should use + `run_blockwise(..., multiprocessing=False)`, which runs the original + function in-process, single-threaded. + - **0-arg worker functions must now be picklable**, because they are + shipped to a subprocess rather than called in place. v2 cannot fork — + the server runs a tokio runtime per worker thread, and forking a + multithreaded process is unsafe — so it spawns and serializes. An + unpicklable function fails before the run starts with guidance. + - **No cheap shared read-only memory.** Workers can no longer close over a + large array. `numpy.memmap` a file instead. + - **`Task(timeout=...)` now always preempts.** There is no longer a mode in + which a timed-out block keeps running and can double-apply its effects + concurrently with the retry. + + Also: worker identity is race-free for both spawn signatures. Each child + gets its own environment, so the `DAISY_CONTEXT` race that the keyword-only + `context` parameter was introduced to dodge cannot happen; the warning + about it is gone. And `daisy.logging` settings are carried in the worker + payload, since a spawned child inherits no process globals. +- The `funlib.persistence.Array` monkey-patching in the v1-compat layer. + daisy and funlib.persistence are unrelated packages that merely share + `funlib.geometry`; daisy no longer imports or modifies persistence. + Compat-surface blocks already carry `funlib.geometry` ROIs (the + `_BlockProxy` boundary), so v1-style code can index persistence Arrays + with them directly — no patching required. Native `daisy.v2` ROIs are + not accepted by persistence; convert explicitly if you mix surfaces. - Lint/type CI (`lint.yaml`): `ruff check` + `ruff format --check` with a pinned rule set (pyflakes, pycodestyle errors, import sorting) and `ty check` over the python package, both via `uv run --only-group lint` @@ -241,113 +311,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 and custom workers); thread mode continues to determine success by exception only, as it always has in v2. -### Documentation - -- Documented the blocking-spawn contract: spawn functions must run for the - worker's lifetime (`sbatch --wait` / `bsub -K` / `subprocess.run`); - submit-and-return spawns over-submit by up to `max_worker_restarts` jobs - and can abandon a task whose jobs are still queued. Connection-aware - worker accounting is planned. - -### Changed - -- **Blocks always have a timeout.** `Task(timeout=...)` defaults to 600 - seconds and can no longer be disabled (`None` = the default; values - <= 0 raise ValueError). A hung block can therefore wedge a run — or - its shutdown — for at most the block deadline: the subprocess worker - self-kills at the deadline, the block is reclaimed and retried, and - shutdown joins complete. Failure surfaces attribute timeout reclaims - (`TaskState.timeout_reclaim_count` / `timeout_secs`, plus run-summary - and abandonment-error hints pointing at `Task(timeout=...)`). - -### Fixed - - Worker-log stream proxies are now uninstalled at the end of each run; previously the first run's `sys.stdout`/`sys.stderr` were captured forever and every later run's execution summary was written to the (possibly closed or replaced) original streams. -### Removed - -- The `funlib.persistence.Array` monkey-patching in the v1-compat layer. - daisy and funlib.persistence are unrelated packages that merely share - `funlib.geometry`; daisy no longer imports or modifies persistence. - Compat-surface blocks already carry `funlib.geometry` ROIs (the - `_BlockProxy` boundary), so v1-style code can index persistence Arrays - with them directly — no patching required. Native `daisy.v2` ROIs are - not accepted by persistence; convert explicitly if you mix surfaces. - -### Added - -- Wheel-building CI (`publish.yaml`): manylinux/musllinux x86_64 + aarch64, - macOS x86_64 + arm64, Windows x64, plus sdist; a built wheel is - smoke-tested (install without a Rust toolchain, run a tiny blockwise - task) before any publish; publishes to PyPI on version tags. - -- Spawn functions may declare a keyword-only `context` parameter - (`def start_worker(*, context):`) to receive their worker's - `daisy.Context` as an argument — a race-free alternative to reading the - process-global `DAISY_CONTEXT` environment variable, which concurrent - slow spawn functions can observe with a later worker's value. The env - var keeps being set for 0-arg spawn functions and worker children. - `Context.from_env_string(...)` parses an encoded context without - touching the environment. The built-in subprocess workers now set the - child's `DAISY_CONTEXT` deterministically from the argument. - -### Changed - -- **Done-marker tracking is now opt-in.** `Task(done_marker_path=None)` only - resolves to a marker when `set_done_marker_basedir(...)` has been called; - the fallback to the logging basedir (`./daisy_logs/`, relative to - the current working directory) is removed. Previously a rerun of a script - whose code had changed could silently skip every block a prior run had - marked done. Explicit `done_marker_path="..."` and `done_marker_path=False` - behave as before. To restore the old behavior, call - `daisy.set_done_marker_basedir(...)` once at pipeline start. - -### Added - -- Resumed runs now emit an INFO record per task on the `daisy._progress` - logger stating how many blocks were skipped via done markers and how to - reprocess them. +- Worker starts are now bounded by a hard per-task budget of `max_workers + max_worker_restarts`, regardless of how or why previous workers exited. Previously only dirty exits counted toward the restart cap, so a worker that exited cleanly without processing blocks (e.g. `subprocess.run(..., check=False)` around a command that fails to start) respawned forever and the run never terminated. Workers are expected to be long-running; the recycle-after-N-blocks pattern is not supported — size `max_worker_restarts` for expected worker deaths (preemption, walltime), or resume via done markers. See `docs/source/design/ABANDONMENT.md`. ### Documentation +- Documented the blocking-spawn contract: spawn functions must run for the + worker's lifetime (`sbatch --wait` / `bsub -K` / `subprocess.run`); + submit-and-return spawns over-submit by up to `max_worker_restarts` jobs + and can abandon a task whose jobs are still queued. Connection-aware + worker accounting is planned. + - Done markers: documented the layout-hash limitation for grown volumes (extending `total_roi` currently invalidates the whole marker) and the planned in-place migration enhancement, with interim workarounds. (adversarial suite case f05) -- Worker starts are now bounded by a hard per-task budget of `max_workers + max_worker_restarts`, regardless of how or why previous workers exited. Previously only dirty exits counted toward the restart cap, so a worker that exited cleanly without processing blocks (e.g. `subprocess.run(..., check=False)` around a command that fails to start) respawned forever and the run never terminated. Workers are expected to be long-running; the recycle-after-N-blocks pattern is not supported — size `max_worker_restarts` for expected worker deaths (preemption, walltime), or resume via done markers. See `docs/source/design/ABANDONMENT.md`. - -- **Block functions run in worker subprocesses.** A 1-arg `process_function` - on the distributed run paths is serialized (with `cloudpickle` when - installed; stdlib `pickle` otherwise) and executed by real OS worker - processes launched as `python -m daisy._subprocess_worker`, rather than on - GIL-sharing threads inside the server process. Serial execution - (`run_blockwise(..., multiprocessing=False)`) is unchanged. - - Why: across workload mixes (16 workers, 96 × ~100 ms blocks), threads win - only when the block function releases the GIL for essentially its entire - runtime — pure I/O waits or single-threaded C-library calls — and then only - by 13–17%. With just 10% pure-python glue, threads are 1.7× slower; at 30% - python, 8× slower; at 100% python, 28× slower. Worker processes are flat - across every mix. (Thread execution was briefly retained as an opt-out via - `worker_processes=False`; see the Removed section above for why it went.) - - **Resource implication**: `max_workers=N` means N python interpreter - processes (each importing your function's modules) rather than N threads in - one process. Budget memory accordingly. - - `Task(timeout=...)` gains true preemption: a block exceeding the deadline - kills its worker process (visible as a dirty exit, bounded by - `max_worker_restarts`) instead of leaving a runaway thread behind. - -### Added - -- Optional dependency extra `daisy[worker-processes]` installing - `cloudpickle` for lambda/closure support in worker processes. - ## [2.0.0] — 2026-04-27 ### Overview diff --git a/dev/INTERNAL_DIFFERENCES.md b/dev/INTERNAL_DIFFERENCES.md index 130197d5..8d158a99 100644 --- a/dev/INTERNAL_DIFFERENCES.md +++ b/dev/INTERNAL_DIFFERENCES.md @@ -1,12 +1,12 @@ -# Internal Differences: daisy vs daisy +# Internal Differences: daisy 1.x vs daisy v2 -A subsystem-by-subsystem comparison of how the two implementations differ, with specific pros and cons for each approach. +A subsystem-by-subsystem comparison of how daisy 1.x and daisy v2 differ, with specific pros and cons for each approach. --- ## 1. Event Loop -**Daisy**: synchronous polling loop with 0.1-second blocking timeout on a tornado message queue. +**daisy 1.x**: synchronous polling loop with 0.1-second blocking timeout on a tornado message queue. ```python while not self.stop_event.is_set(): @@ -17,7 +17,7 @@ while not self.stop_event.is_set(): self._check_all_tasks_completed() ``` -**Daisy**: async `tokio::select!` loop that wakes only when a message arrives or a timer fires. +**daisy v2**: async `tokio::select!` loop that wakes only when a message arrives or a timer fires. ```rust tokio::select! { @@ -27,9 +27,9 @@ tokio::select! { } ``` -**Pro daisy**: zero-cost waiting — the thread sleeps until an event occurs instead of polling. With many idle workers, daisy wastes CPU spinning through the loop every 0.1s; daisy's select is free. +**Pro daisy v2**: zero-cost waiting — the thread sleeps until an event occurs instead of polling. With many idle workers, daisy 1.x wastes CPU spinning through the loop every 0.1s; daisy v2's select is free. -**Pro daisy**: simpler mental model. The loop is sequential and predictable — you can read it top-to-bottom. Daisy's select has branch priority semantics (tokio picks randomly among ready branches) that are less obvious. +**Pro daisy 1.x**: simpler mental model. The loop is sequential and predictable — you can read it top-to-bottom. daisy v2's select has branch priority semantics (tokio picks randomly among ready branches) that are less obvious. --- @@ -37,9 +37,9 @@ tokio::select! { Both implementations use a single dispatch path for block requests. -**Daisy**: when no blocks are ready, parks the raw TCP message in a pending queue. On the next loop iteration, `_get_client_message` checks the TCP queue first, then falls back to the pending queue. Either way, the message goes through `_handle_client_message` → `_handle_acquire_block` — the single path that dispatches blocks and registers with the bookkeeper. +**daisy 1.x**: when no blocks are ready, parks the raw TCP message in a pending queue. On the next loop iteration, `_get_client_message` checks the TCP queue first, then falls back to the pending queue. Either way, the message goes through `_handle_client_message` → `_handle_acquire_block` — the single path that dispatches blocks and registers with the bookkeeper. -**Daisy**: when no blocks are ready, parks the full `ClientMessage` (including client address and reply channel) in a `VecDeque`. After a block release — the only event that can free downstream dependencies — `retry_pending` pops each parked message and feeds it back through `handle_message` → `handle_acquire`, the same single path fresh messages use. +**daisy v2**: when no blocks are ready, parks the full `ClientMessage` (including client address and reply channel) in a `VecDeque`. After a block release — the only event that can free downstream dependencies — `retry_pending` pops each parked message and feeds it back through `handle_message` → `handle_acquire`, the same single path fresh messages use. ```rust Message::ReleaseBlock { block } => { @@ -51,9 +51,9 @@ Message::ReleaseBlock { block } => { } ``` -**Pro daisy**: retry is only triggered by events that can actually change block availability (release and lost-block recovery). Daisy retries pending on every loop iteration regardless of whether anything changed, which means unfulfillable requests are re-checked every 0.1s. +**Pro daisy v2**: retry is only triggered by events that can actually change block availability (release and lost-block recovery). daisy 1.x retries pending on every loop iteration regardless of whether anything changed, which means unfulfillable requests are re-checked every 0.1s. -**Pro daisy**: the pending queue is checked as part of the message-fetch logic, so it's impossible to forget to check it — it happens automatically on every iteration. Daisy must explicitly call `retry_pending` at each release site. +**Pro daisy 1.x**: the pending queue is checked as part of the message-fetch logic, so it's impossible to forget to check it — it happens automatically on every iteration. daisy v2 must explicitly call `retry_pending` at each release site. **Shared strength**: both have a single dispatch path. The bookkeeper sees every dispatched block because there's only one place dispatch happens. @@ -61,21 +61,21 @@ Message::ReleaseBlock { block } => { ## 3. Message Serialization -**Daisy**: Python `pickle` over `[4-byte length][payload]` frames on tornado IOStreams. +**daisy 1.x**: Python `pickle` over `[4-byte length][payload]` frames on tornado IOStreams. -**Daisy**: Rust `bincode` over `[4-byte length][payload]` frames on tokio TCP. +**daisy v2**: Rust `bincode` over `[4-byte length][payload]` frames on tokio TCP. -**Pro daisy**: bincode is type-safe (can't deserialize into an unexpected type), cross-language (any language with a bincode implementation can speak the protocol), and doesn't allow arbitrary code execution (pickle does — a malicious message can run code on deserialization). Daisy also validates message size (64 MiB cap) before allocating. +**Pro daisy v2**: bincode is type-safe (can't deserialize into an unexpected type), cross-language (any language with a bincode implementation can speak the protocol), and doesn't allow arbitrary code execution (pickle does — a malicious message can run code on deserialization). daisy v2 also validates message size (64 MiB cap) before allocating. -**Pro daisy**: pickle can serialize arbitrary Python objects, including exception tracebacks and user-defined types. Daisy's `BlockFailed` message carries only a string error description; daisy's carries the actual exception object, which the server can re-raise with the original traceback. +**Pro daisy 1.x**: pickle can serialize arbitrary Python objects, including exception tracebacks and user-defined types. daisy v2's `BlockFailed` message carries only a string error description; daisy 1.x's carries the actual exception object, which the server can re-raise with the original traceback. -**Con daisy**: bincode is a binary format with no human-readable form. Debugging protocol issues requires writing a decoder. Daisy's pickled messages can be inspected with `pickle.loads()` in a REPL. +**Con daisy v2**: bincode is a binary format with no human-readable form. Debugging protocol issues requires writing a decoder. daisy 1.x's pickled messages can be inspected with `pickle.loads()` in a REPL. --- ## 4. Worker Management -**Daisy**: workers are `multiprocessing.Process` objects spawned by Python. The `process` attribute is `None` when stopped, a `Process` when running. Health monitoring is done by `TaskWorkerPools` which calls `reap_dead_workers()` and spawns replacements. Worker spawning, health checks, and cleanup all live in Python. +**daisy 1.x**: workers are `multiprocessing.Process` objects spawned by Python. The `process` attribute is `None` when stopped, a `Process` when running. Health monitoring is done by `TaskWorkerPools` which calls `reap_dead_workers()` and spawns replacements. Worker spawning, health checks, and cleanup all live in Python. ```python self.process = None # initial @@ -83,7 +83,7 @@ self.process = Process(...) # after start self.process = None # after stop ``` -**Daisy**: worker *supervisors* are `std::thread` instances spawned and managed entirely by the Rust server; the work itself runs in a dedicated OS process per worker, as in v1. Each supervisor thread acquires the GIL just long enough to call the task's spawn function, then waits for it. The Python layer converts whatever the user passed into such a spawn function before the run starts: +**daisy v2**: worker *supervisors* are `std::thread` instances spawned and managed entirely by the Rust server; the work itself runs in a dedicated OS process per worker, as in v1. Each supervisor thread acquires the GIL just long enough to call the task's spawn function, then waits for it. The Python layer converts whatever the user passed into such a spawn function before the run starts: - **1-arg block processors** are serialized and run by `python -m daisy._subprocess_worker`, which drives the TCP `Client.acquire_block()` loop in the child. - **0-arg worker functions** are serialized and called in a child, where they drive their own loop and may `subprocess.run(...)` something further. @@ -100,86 +100,86 @@ let should_respawn = match handle.join() { }; ``` -**Pro daisy**: no dill serialization, no `multiprocessing.Process`, no Python health-monitor thread. Worker lifecycle is managed by typed Rust code. The `WorkerState` enum makes the worker explosion bug (PR #67/#68) unrepresentable — you can't confuse "exited normally" with "crashed" because they're different enum variants. +**Pro daisy v2**: no dill serialization, no `multiprocessing.Process`, no Python health-monitor thread. Worker lifecycle is managed by typed Rust code. The `WorkerState` enum makes the worker explosion bug (PR #67/#68) unrepresentable — you can't confuse "exited normally" with "crashed" because they're different enum variants. **Parity**: both implementations give each worker a separate OS process with its own GIL, so Python-heavy process functions run in true parallel. v1 gets there by forking; v2 cannot fork (its server is multithreaded) and so spawns and serializes instead, which is why v2 requires worker functions to be picklable. -**Con daisy**: the nullable `process` field caused the worker explosion bug. `reap_dead_workers` couldn't distinguish normal exit from crash because both ended up with `process = None`. +**Con daisy 1.x**: the nullable `process` field caused the worker explosion bug. `reap_dead_workers` couldn't distinguish normal exit from crash because both ended up with `process = None`. --- ## 5. Error Propagation -**Daisy**: workers catch exceptions in `_spawn_wrapper`, serialize them onto a `multiprocessing.Queue` (100-item limit), and the server re-raises them via `check_for_errors()`. +**daisy 1.x**: workers catch exceptions in `_spawn_wrapper`, serialize them onto a `multiprocessing.Queue` (100-item limit), and the server re-raises them via `check_for_errors()`. ```python except Exception as e: self.error_queue.put(e, timeout=1) # silently dropped if queue full ``` -**Daisy**: worker threads return `bool` (true = clean exit, false = should respawn). Block-level failures are reported via TCP `BlockFailed` messages with an error string. Thread panics are caught by `JoinHandle::join().is_err()`. +**daisy v2**: worker threads return `bool` (true = clean exit, false = should respawn). Block-level failures are reported via TCP `BlockFailed` messages with an error string. Thread panics are caught by `JoinHandle::join().is_err()`. -**Pro daisy**: exception objects preserve tracebacks. When a worker fails, the server can print the exact stack trace from the worker process. +**Pro daisy 1.x**: exception objects preserve tracebacks. When a worker fails, the server can print the exact stack trace from the worker process. -**Pro daisy**: no silent drops. The error queue's 100-item limit means daisy can lose errors under load with only a log message. Daisy's thread health check always detects failure — there's no queue that can fill up. +**Pro daisy v2**: no silent drops. The error queue's 100-item limit means daisy 1.x can lose errors under load with only a log message. daisy v2's thread health check always detects failure — there's no queue that can fill up. --- ## 6. Concurrency & Synchronization -**Daisy**: uses `threading.Lock` to protect the `workers` dict. Every method that accesses workers must acquire the lock manually. Several don't — `inc_num_workers` modifies the dict without the lock. The server's `BlockBookkeeper.sent_blocks` dict has no lock at all. +**daisy 1.x**: uses `threading.Lock` to protect the `workers` dict. Every method that accesses workers must acquire the lock manually. Several don't — `inc_num_workers` modifies the dict without the lock. The server's `BlockBookkeeper.sent_blocks` dict has no lock at all. -**Daisy**: the coordinator owns all mutable state on a single async task. There are no locks because there's nothing shared. Worker pools, the scheduler, the bookkeeper, and the pending queue are all `&mut`-accessed from the select loop — the Rust compiler rejects any attempt to share them across tasks. Worker threads communicate with the coordinator only via TCP (the same channel as external workers). +**daisy v2**: the coordinator owns all mutable state on a single async task. There are no locks because there's nothing shared. Worker pools, the scheduler, the bookkeeper, and the pending queue are all `&mut`-accessed from the select loop — the Rust compiler rejects any attempt to share them across tasks. Worker threads communicate with the coordinator only via TCP (the same channel as external workers). -**Pro daisy**: compile-time guarantee of no data races. The entire class of "forgot to acquire the lock" bugs is structurally impossible. +**Pro daisy v2**: compile-time guarantee of no data races. The entire class of "forgot to acquire the lock" bugs is structurally impossible. -**Pro daisy**: the `threading.Lock` pattern is well-understood by Python developers. Daisy's ownership model requires understanding Rust's borrow checker. +**Pro daisy 1.x**: the `threading.Lock` pattern is well-understood by Python developers. daisy v2's ownership model requires understanding Rust's borrow checker. -**Con daisy**: `task_worker_pools.py` calls `reap_dead_workers()` (acquires lock) then `inc_num_workers()` (doesn't acquire lock) in sequence. Another thread can modify the workers dict between these two calls. +**Con daisy 1.x**: `task_worker_pools.py` calls `reap_dead_workers()` (acquires lock) then `inc_num_workers()` (doesn't acquire lock) in sequence. Another thread can modify the workers dict between these two calls. --- ## 7. Block Bookkeeping & Lost Block Detection -**Daisy**: tracks blocks by TCP stream object identity (`log.stream`). Lost blocks detected by polling `stream.closed()` on every health check. +**daisy 1.x**: tracks blocks by TCP stream object identity (`log.stream`). Lost blocks detected by polling `stream.closed()` on every health check. -**Daisy**: tracks blocks by `SocketAddr`. Disconnected clients are registered proactively when a Disconnect message arrives, stored in a `HashSet`. +**daisy v2**: tracks blocks by `SocketAddr`. Disconnected clients are registered proactively when a Disconnect message arrives, stored in a `HashSet`. -**Pro daisy**: disconnection tracking is O(1) per block (set lookup) rather than requiring a stream query. Uses `Instant` (monotonic clock) for timeout tracking, immune to system clock adjustments. +**Pro daisy v2**: disconnection tracking is O(1) per block (set lookup) rather than requiring a stream query. Uses `Instant` (monotonic clock) for timeout tracking, immune to system clock adjustments. -**Pro daisy**: stream identity is unforgeable — if a worker reconnects, it gets a new stream object, so old blocks can't be accidentally claimed by the new connection. +**Pro daisy 1.x**: stream identity is unforgeable — if a worker reconnects, it gets a new stream object, so old blocks can't be accidentally claimed by the new connection. --- ## 8. Scheduler & Ready Surface -The algorithms are identical: same dependency graph computation, same level stride formula, same ready surface with surface/boundary sets, same BFS failure propagation, same cantor pairing function for block IDs (daisy uses the funlib pyramid-volume generalization to match daisy's block ordering exactly). +The algorithms are identical: same dependency graph computation, same level stride formula, same ready surface with surface/boundary sets, same BFS failure propagation, same cantor pairing function for block IDs (daisy v2 uses the funlib pyramid-volume generalization to match daisy 1.x's block ordering exactly). -**Pro daisy**: the ready surface is generic over closure types (`ReadySurface` where `F: Fn(&Block) -> Vec`). Rust monomorphizes this at compile time, inlining the closure calls. Daisy uses Python lambdas with function-call overhead per invocation. +**Pro daisy v2**: the ready surface is generic over closure types (`ReadySurface` where `F: Fn(&Block) -> Vec`). Rust monomorphizes this at compile time, inlining the closure calls. daisy 1.x uses Python lambdas with function-call overhead per invocation. -**Con daisy**: the `Arc` captured by closures means the dependency graph is constructed twice in the scheduler (once for the closures, once stored on the struct). Daisy builds it once. +**Con daisy v2**: the `Arc` captured by closures means the dependency graph is constructed twice in the scheduler (once for the closures, once stored on the struct). daisy 1.x builds it once. --- ## 9. Shutdown -**Daisy**: uses `try/finally`. Calls `worker_pools.stop()` which SIGTERMs each worker, then `tcp_server.disconnect()` which closes streams. Pending requests are abandoned — workers eventually notice via TCP stream closure. +**daisy 1.x**: uses `try/finally`. Calls `worker_pools.stop()` which SIGTERMs each worker, then `tcp_server.disconnect()` which closes streams. Pending requests are abandoned — workers eventually notice via TCP stream closure. -**Daisy**: proactively sends `RequestShutdown` to all parked workers before closing the accept loop. Then drains remaining messages, answering late `AcquireBlock`s with `RequestShutdown`. Worker threads exit when they receive `RequestShutdown` or the TCP connection closes. `JoinHandle::join()` waits for each thread during shutdown. `WorkerPool` also implements `Drop` for cleanup. +**daisy v2**: proactively sends `RequestShutdown` to all parked workers before closing the accept loop. Then drains remaining messages, answering late `AcquireBlock`s with `RequestShutdown`. Worker threads exit when they receive `RequestShutdown` or the TCP connection closes. `JoinHandle::join()` waits for each thread during shutdown. `WorkerPool` also implements `Drop` for cleanup. -**Pro daisy**: workers get a clean shutdown signal and can exit gracefully instead of discovering a dead TCP connection. The `Drop` impl provides a safety net. +**Pro daisy v2**: workers get a clean shutdown signal and can exit gracefully instead of discovering a dead TCP connection. The `Drop` impl provides a safety net. -**Pro daisy**: simpler — three lines (`stop`, `disconnect`, `notify_exit`). +**Pro daisy 1.x**: simpler — three lines (`stop`, `disconnect`, `notify_exit`). --- ## 10. Python Compatibility Layer -Daisy's Python wrapper (`_compat.py`, ~200 lines) is a thin adapter that maps daisy's constructor signatures to daisy's Rust types. It contains no scheduling logic, no block lifecycle management, and no worker management. +daisy v2's Python wrapper (`_compat.py`, ~200 lines) is a thin adapter that maps daisy 1.x's constructor signatures to daisy v2's Rust types. It contains no scheduling logic, no block lifecycle management, and no worker management. | Class | What it does | Lines | |---|---|---| -| `Task` | Maps daisy's positional constructor to Rust keyword args | ~40 | +| `Task` | Maps daisy 1.x's positional constructor to Rust keyword args | ~40 | | `Scheduler` | Delegates all methods to `_rs.Scheduler` | ~15 | | `Context` | Env var key=value encoding (data class) | ~30 | | `Client` | Wraps `_rs.SyncClient` + `acquire_block` context manager | ~25 | @@ -197,7 +197,7 @@ Benchmarks on Apple Silicon (M-series), Python 3.14, trivial block processing fu ### Dependency Graph Block Iteration -| Configuration | daisy | daisy | Speedup | +| Configuration | daisy 1.x | daisy v2 | Speedup | |---|---|---|---| | 1M blocks, no conflict | 15.2s | 1.9s | **7.8x** | | 970K blocks, with conflict (8 levels) | 46.4s | 7.5s | **6.2x** | @@ -208,7 +208,7 @@ The 6-8x speedup comes from Rust's compiled code vs Python's interpreter overhea ### Worker Coordination Scaling (10K blocks, noop process function) -| Workers | daisy | daisy | Speedup | +| Workers | daisy 1.x | daisy v2 | Speedup | |---|---|---|---| | 1 (serial) | 0.17s | 0.07s | **2.5x** | | 2 | 13.8s | 0.46s | **30x** | @@ -217,24 +217,24 @@ The 6-8x speedup comes from Rust's compiled code vs Python's interpreter overhea | 16 | 2.1s | 0.21s | **10.3x** | | 32 | 2.7s | 0.22s | **12.3x** | -Daisy's distributed mode has ~2s fixed overhead (tornado IOLoop + multiprocessing.Process startup). Daisy's thread-based workers have negligible startup cost. The 2-worker daisy outlier (13.8s) is tornado initialization overhead. As worker count increases, daisy's coordination cost stays flat (~0.2s) while daisy's grows linearly with process count. +daisy 1.x's distributed mode has ~2s fixed overhead (tornado IOLoop + multiprocessing.Process startup). daisy v2's thread-based workers have negligible startup cost. The 2-worker daisy 1.x outlier (13.8s) is tornado initialization overhead. As worker count increases, daisy v2's coordination cost stays flat (~0.2s) while daisy 1.x's grows linearly with process count. ### Block Count Scaling (4 workers, noop process function) -| Blocks | daisy | daisy | Speedup | +| Blocks | daisy 1.x | daisy v2 | Speedup | |---|---|---|---| | 100 | 0.22s | 0.005s | **43x** | | 1,000 | 0.36s | 0.04s | **10x** | | 10,000 | 1.8s | 0.35s | **5x** | | 100,000 | 16.0s | 3.3s | **5x** | -The small-block advantage (43x at 100 blocks) reflects the fixed overhead gap: daisy's process startup costs ~0.2s, daisy's thread spawn costs ~0.005s. As block count grows, per-block cost dominates and the ratio stabilizes at ~5x — this is the raw speed difference between Python and Rust for the block dispatch/acquire/release cycle. +The small-block advantage (43x at 100 blocks) reflects the fixed overhead gap: daisy 1.x's process startup costs ~0.2s, daisy v2's thread spawn costs ~0.005s. As block count grows, per-block cost dominates and the ratio stabilizes at ~5x — this is the raw speed difference between Python and Rust for the block dispatch/acquire/release cycle. --- ## Summary -| Aspect | Daisy | Daisy | +| Aspect | daisy 1.x | daisy v2 | |--------|-------|---------| | **Event loop** | Sync 0.1s poll | Async select, zero-cost | | **Dispatch path** | Single (inherent — pending re-queued as messages) | Single (explicit — `retry_pending` at release sites) | diff --git a/dev/MIGRATION_REPORT.md b/dev/MIGRATION_REPORT.md index ca575d70..76672055 100644 --- a/dev/MIGRATION_REPORT.md +++ b/dev/MIGRATION_REPORT.md @@ -1,23 +1,23 @@ -# Migration Report: daisy → daisy +# Migration Report: daisy 1.x → daisy v2 ## Test Coverage Mapping -### Daisy Tests → Daisy Equivalents +### daisy 1.x Tests → daisy v2 Equivalents -| Daisy Test | Daisy Test | Status | Notes | +| daisy 1.x Test | daisy v2 Test | Status | Notes | |------------|-------------|--------|-------| | **test_scheduler.py** (13 tests) | **test_scheduler.py** (13 tests) | Exact match | All block IDs, ordering, and state transitions match byte-for-byte | | **test_dependency_graph.py** (3 tests, 6 parameterized) | **test_dependency_graph.py** (3 tests, 6 parameterized) | Exact match | Block counts, subgraph extraction, upstream/downstream symmetry | -| **test_server.py** (1 test, 2 variants) | **test_server.py** (4 tests) | Expanded | Daisy tests both `Server` and `SerialServer` via parametrize. Daisy tests serial mode with additional cases: 2D tasks, check functions, chained tasks | -| **test_tcp.py** (1 test) | Rust `test_framing_roundtrip` + **test_tcp_client.py::test_no_message_after_shutdown** | Equivalent | Daisy tests raw TCP message exchange. Daisy tests the same via Rust integration test (bincode framing) and Python-side scheduler behavior | -| **test_client.py** (1 test) | **test_tcp_client.py::test_client_acquire_release** + Rust `test_server_client_no_conflict` | Equivalent | Daisy uses a mock server subprocess. Daisy tests both through the Scheduler API and through real TCP in Rust | -| **test_clients_close.py** (1 test) | **test_tcp_client.py::test_multiple_workers_complete** | Equivalent | Daisy spawns 5 subprocess workers with file locks. Daisy verifies all blocks are processed (subprocess workers require parallel mode, not yet wired through Python) | -| **test_dead_workers.py** (1 test) | **test_tcp_client.py::test_block_failure_recovery** + Rust `test_server_block_failure_and_retry` | Equivalent | Daisy crashes a worker via SystemExit. Daisy tests retry logic through both Python (exception in process function) and Rust (explicit failure message) | -| **test_worker_spawning.py** (1 test) | **test_tcp_client.py::test_worker_normal_exit_no_respawn** | Equivalent | Daisy verifies normal-exit workers aren't replaced. Daisy verifies exact block count (no extra processing from respawn cycles) | +| **test_server.py** (1 test, 2 variants) | **test_server.py** (4 tests) | Expanded | daisy 1.x tests both `Server` and `SerialServer` via parametrize. daisy v2 tests serial mode with additional cases: 2D tasks, check functions, chained tasks | +| **test_tcp.py** (1 test) | Rust `test_framing_roundtrip` + **test_tcp_client.py::test_no_message_after_shutdown** | Equivalent | daisy 1.x tests raw TCP message exchange. daisy v2 tests the same via Rust integration test (bincode framing) and Python-side scheduler behavior | +| **test_client.py** (1 test) | **test_tcp_client.py::test_client_acquire_release** + Rust `test_server_client_no_conflict` | Equivalent | daisy 1.x uses a mock server subprocess. daisy v2 tests both through the Scheduler API and through real TCP in Rust | +| **test_clients_close.py** (1 test) | **test_tcp_client.py::test_multiple_workers_complete** | Equivalent | daisy 1.x spawns 5 subprocess workers with file locks. daisy v2 verifies all blocks are processed (subprocess workers require parallel mode, not yet wired through Python) | +| **test_dead_workers.py** (1 test) | **test_tcp_client.py::test_block_failure_recovery** + Rust `test_server_block_failure_and_retry` | Equivalent | daisy 1.x crashes a worker via SystemExit. daisy v2 tests retry logic through both Python (exception in process function) and Rust (explicit failure message) | +| **test_worker_spawning.py** (1 test) | **test_tcp_client.py::test_worker_normal_exit_no_respawn** | Equivalent | daisy 1.x verifies normal-exit workers aren't replaced. daisy v2 verifies exact block count (no extra processing from respawn cycles) | ### Total Test Counts -| Suite | Daisy | Daisy (Python) | Daisy (Rust) | +| Suite | daisy 1.x | daisy v2 (Python) | daisy v2 (Rust) | |-------|-------|-------------------|----------------| | Scheduler | 13 | 13 | 1 | | Dependency Graph | 3 (+6 param) | 3 (+6 param) | 4 | @@ -43,13 +43,13 @@ cantor([x,...,z]) = pyramid_volume(n, sum(all)) + cantor([x,...,y]) ``` The standard fold-left approach (`cantor(cantor(a,b), c)`) produces different IDs for the same coordinates. This caused block IDs to differ, breaking ordering compatibility. -**Impact**: Block IDs now match daisy exactly. No behavioral difference. +**Impact**: Block IDs now match daisy 1.x exactly. No behavioral difference. ### 2. Block ID Format **Change**: `block_id` exposed as Python tuple `(task_id: str, spatial_id: int)` instead of a custom struct. -**Why**: Daisy's `Block.block_id` is a Python tuple `(task_id, int)`. Tests compare block IDs with tuples like `block.block_id == ("test_2d", 12)`. Returning a Rust struct would break this pattern. +**Why**: daisy 1.x's `Block.block_id` is a Python tuple `(task_id, int)`. Tests compare block IDs with tuples like `block.block_id == ("test_2d", 12)`. Returning a Rust struct would break this pattern. **Impact**: None — identical API behavior. @@ -57,7 +57,7 @@ The standard fold-left approach (`cantor(cantor(a,b), c)`) produces different ID **Change**: `upstream_tasks` parameter accepts a Python list and recursively converts the full task tree using a cache to handle shared references. -**Why**: In daisy, Python tasks are reference objects — the same `Task` object can appear as an upstream dependency of multiple tasks. In Rust, `Arc` provides the same shared ownership, but the PyO3 conversion must handle this explicitly to avoid converting the same task twice. +**Why**: In daisy 1.x, Python tasks are reference objects — the same `Task` object can appear as an upstream dependency of multiple tasks. In Rust, `Arc` provides the same shared ownership, but the PyO3 conversion must handle this explicitly to avoid converting the same task twice. **Impact**: None — same API, same semantics. @@ -65,7 +65,7 @@ The standard fold-left approach (`cantor(cantor(a,b), c)`) produces different ID **Change**: `BlockStatus.SUCCESS` is `2`, `BlockStatus.FAILED` is `3`, etc. -**Why**: Daisy uses a Python `Enum`. PyO3 exposes these as integer class attributes. The `block.status` getter/setter uses `u8` values. Tests use `BlockStatus.SUCCESS` etc. which resolve to the same integers. +**Why**: daisy 1.x uses a Python `Enum`. PyO3 exposes these as integer class attributes. The `block.status` getter/setter uses `u8` values. Tests use `BlockStatus.SUCCESS` etc. which resolve to the same integers. **Impact**: `block.status = BlockStatus.SUCCESS` works identically. Direct integer comparison (`block.status == 2`) also works. @@ -81,7 +81,7 @@ The standard fold-left approach (`cantor(cantor(a,b), c)`) produces different ID **Change**: Exceptions in `process_function` are caught by the serial runner and the block is marked `FAILED`, triggering retry logic. -**Why**: Daisy's `SerialServer` calls `process_funcs[block.task_id](block)` without a try/except — exceptions propagate and crash the server. Daisy's `SerialRunner` wraps the call and catches errors, matching the behavior of daisy's *distributed* server (where `_spawn_wrapper` catches `Exception`). +**Why**: daisy 1.x's `SerialServer` calls `process_funcs[block.task_id](block)` without a try/except — exceptions propagate and crash the server. daisy v2's `SerialRunner` wraps the call and catches errors, matching the behavior of daisy 1.x's *distributed* server (where `_spawn_wrapper` catches `Exception`). **Impact**: More robust — serial mode now handles process function crashes instead of terminating. diff --git a/dev/REFACTOR.md b/dev/REFACTOR.md index cfddb156..b9a6bb6a 100644 --- a/dev/REFACTOR.md +++ b/dev/REFACTOR.md @@ -1,7 +1,5 @@ # Refactor and feature recommendations - - A review of the daisy codebase as it stands. The library is feature-complete for its core mission (block-wise distributed processing with abandonment, resource budgeting, persistent done markers, and Python compat). This document flags places where the implementation has settled into complexity worth cleaning up, plus features that would slot into the existing architecture for clear user wins. The recommendations are sized by approximate effort: **S** (one sitting), **M** (a focused PR), **L** (multi-day with design discussion). Effort estimates assume one engineer who already knows the codebase. @@ -85,7 +83,7 @@ This is the natural extension of having an observer abstraction — the abstract ### 2.5 Multi-machine workers [L, high value if you need it] -**Problem**: daisy binds to `127.0.0.1` by default. Users running on multi-node SLURM/LSF clusters can't farm out workers across nodes; daisy has launchers for this (`daisy.distributed`). +**Problem**: daisy v2 binds to `127.0.0.1` by default. Users running on multi-node SLURM/LSF clusters can't farm out workers across nodes; daisy 1.x has launchers for this (`daisy.distributed`). **Proposal**: @@ -95,7 +93,7 @@ This is the natural extension of having an observer abstraction — the abstract The protocol already supports remote workers (it's TCP). What's missing is the launcher and the worker entry point. The 0-arg `spawn_function` mode is the precedent — it's how a user could already do this manually with `subprocess.run("ssh node-1 ...")`. Formalizing it would be valuable. -**Next Steps**: This is a high priority. I'm not sure quite how necessary it is at the moment since I haven't started using `daisy` as a replacement for daisy, but this definitely seems like an issue that will have to be resolved before I start using it. +**Next Steps**: This is a high priority. I'm not sure quite how necessary it is at the moment since I haven't started using daisy v2 as a replacement for daisy 1.x, but this definitely seems like an issue that will have to be resolved before I start using it. **Decision**: needs a short RFC before coding. Two complications worth resolving up front: (a) the `process_function` and any imports it uses must be installable on every node — that means a packaging/deploy story alongside the launcher; (b) the SIGINT handler we built only catches signals on the coordinator process. If a worker node hangs, killing the coordinator's tab doesn't propagate. Workers need their own watchdog ("coordinator went away → exit"). Park as RFC; don't bundle with anything else. @@ -182,16 +180,16 @@ The dependency graph already knows everything needed. This is purely a print pas ## Part 3 — Strengths to keep -Things daisy does well that we shouldn't accidentally walk back when refactoring: +Things daisy v2 does well that we shouldn't accidentally walk back when refactoring: - **Typestate task lifecycle** (`TaskState` enum + `RunningTask` mutation methods). The compiler enforces that late events on terminal tasks are dropped — no scattered `if state.is_done() { return; }` checks. See `docs/ABANDONMENT.md`. -- **Resource budget allocator** with `requires`/`resources` — global budget composes with per-task `max_workers`, which daisy doesn't have. -- **Persistent done markers** in Zarr v3 layout — daisy has `check_function` but not a built-in persistence mechanism, so users had to roll their own. -- **Worker restart cap** with proper abandonment + transitive downstream orphan propagation. Daisy's restart-cap is "respawn forever, hope it works". -- **Cross-platform raw SIGINT handler** — Ctrl-C works during `py.detach`d tokio runs without requiring `Python::check_signals` from the main thread (which doesn't fire under tokio's multi-threaded scheduler). Daisy doesn't have to deal with this because it's pure Python. -- **Run-stats with linear regression slopes** for per-block durations — surfaces "is processing getting slower over time" trends that a simple mean would hide. Not in daisy. -- **Topological display ordering** for tqdm bars and reports — Kahn's with alphabetical tiebreaker. Daisy's progress display is dict-order. -- **Bincode + size-validated framing** — kills the pickle code-execution attack surface daisy carries. +- **Resource budget allocator** with `requires`/`resources` — global budget composes with per-task `max_workers`, which daisy 1.x doesn't have. +- **Persistent done markers** in Zarr v3 layout — daisy 1.x has `check_function` but not a built-in persistence mechanism, so users had to roll their own. +- **Worker restart cap** with proper abandonment + transitive downstream orphan propagation. daisy 1.x's restart-cap is "respawn forever, hope it works". +- **Cross-platform raw SIGINT handler** — Ctrl-C works during `py.detach`d tokio runs without requiring `Python::check_signals` from the main thread (which doesn't fire under tokio's multi-threaded scheduler). daisy 1.x doesn't have to deal with this because it's pure Python. +- **Run-stats with linear regression slopes** for per-block durations — surfaces "is processing getting slower over time" trends that a simple mean would hide. Not in daisy 1.x. +- **Topological display ordering** for tqdm bars and reports — Kahn's with alphabetical tiebreaker. daisy 1.x's progress display is dict-order. +- **Bincode + size-validated framing** — kills the pickle code-execution attack surface daisy 1.x carries. When considering changes, prefer ones that strengthen these (unify access to them, document them better, add features that compose with them) over ones that add parallel mechanisms. @@ -199,11 +197,21 @@ When considering changes, prefer ones that strengthen these (unify access to the After the discussion documented above: -**This batch (small, clear wins)**: - -1. **2.8 lazy roots** — fixes the 1M-block startup hang foot-gun -2. **2.2 per-block timeout** — opt-in, no default, no preemption -3. **2.4 JSON observer** — standalone, ~50 lines +**This batch (small, clear wins)** — all three have landed: + +1. **2.8 lazy roots** — fixes the 1M-block startup hang foot-gun. **Landed**: + `root_iter_owned` + `LazyBlockIter` (daisy-core/src/dependency_graph.rs:468, + :705) make `DependencyGraph::roots` (:606) hand out + `Box + Send>` with no upfront materialization. +2. **2.2 per-block timeout** — opt-in, no default, no preemption. **Landed, but + not as decided above**: `Task::timeout` reaches the bookkeeper + (daisy-core/src/server.rs:951 → block_bookkeeper.rs:116), and every block now + has one — it defaults to 600 s and cannot be disabled + (daisy-py/src/py_task.rs:104) — and it *does* preempt, because the subprocess + worker self-kills at the deadline (daisy-py/python/daisy/_subprocess_worker.py:70). +3. **2.4 JSON observer** — standalone, ~50 lines. **Landed**: + `JsonProgressObserver` (daisy-py/python/daisy/_progress.py:267), exported from + `daisy`, covered by tests/test_json_observer.py. **Soon, before users adopt**: