Skip to content

fix: blockwise tasks never resume — check_function always raises on daisy v2 - #65

Draft
rhoadesScholar wants to merge 4 commits into
will/rbws-plus-mainfrom
datum/blockwise-resume
Draft

fix: blockwise tasks never resume — check_function always raises on daisy v2#65
rhoadesScholar wants to merge 4 commits into
will/rbws-plus-mainfrom
datum/blockwise-resume

Conversation

@rhoadesScholar

Copy link
Copy Markdown
Contributor

check_function raises on every block, so no block is ever seen as done and every rerun recomputes everything.

Reproduction

One daisy task, 64 blocks, run twice. Second run should skip all 64.

def check_block(block):                                  # volara/blockwise/blockwise.py:204-215
    block_array = open_ds(block_ds_path, mode="r")
    coordinate = (block.write_roi.offset - block_array.offset) / block_array.voxel_size
    ...

task = daisy.Task(..., check_function=check_block)
daisy.run_blockwise([task], multiprocessing=False)       # twice

On main:

run 1: process_function called on 64 blocks
run 2: process_function called on 64 blocks
check received block.write_roi.offset of type: daisy._daisy.Coordinate
check RAISED: TypeError: unsupported operand type(s) for -: 'daisy._daisy.Coordinate' and 'Coordinate'

On this branch:

run 1: process_function called on 64 blocks
run 2: process_function called on 0 blocks
Execution Summary:  blocks 64   completed 64   skipped 64

Cause

daisy v2's compat layer wraps process_function so it receives a funlib-typed Block
(daisy/v1_compat.py:221-225) but does not wrap check_function. check_block therefore gets a
native Block, and blockwise.py:209 evaluates daisy._daisy.Coordinate - funlib.geometry.Coordinate:

>>> daisy.Coordinate((0,0,0)) - funlib.geometry.Coordinate((1,1,1))
TypeError: unsupported operand type(s) for -: 'daisy._daisy.Coordinate' and 'Coordinate'

The scheduler catches a raising check as "not done", so the failure is silent.

Fix

Use daisy's built-in done-markers rather than checking blocks_done.zarr through check_function.
tracking_path is under meta_dir, so drop() still clears the done state and rerun semantics are
unchanged — markers outside meta_dir would survive a drop and skip everything.

blocks_done.zarr is left in place; removing it is a separate change.

Tests

3 failed, 182 passed on both main and this branch — the three test_logging failures are
pre-existing and unrelated.

Related: funkelab/daisy#80 wraps check_function upstream, which fixes released volara versions too.
These are complementary.

sheridana and others added 3 commits July 30, 2026 13:05
Add s3 uri option for lut class
…aisy v2

daisy v2's compat layer wraps `process_function` so it receives a funlib-typed
Block, but not `check_function` (daisy/v1_compat.py:221-225). So `check_block`
gets a native Block and blockwise.py:209 evaluates

    block.write_roi.offset - block_array.offset

as daisy._daisy.Coordinate - funlib.geometry.Coordinate -> TypeError. The
scheduler treats a raising check as "not done", so every block is recomputed on
every rerun, silently.

Use daisy's own done-markers instead. tracking_path lives under meta_dir so
drop() keeps clearing the done state, and rerun semantics are unchanged.
@rhoadesScholar

rhoadesScholar commented Aug 12, 2026

Copy link
Copy Markdown
Contributor Author

Review, rewritten 2026-08-12 — the base blocker is resolved and the conflict is fixed, so this replaces a much longer earlier version.

Conflict fixed (270c31b). Merge, not rebase, so nothing was force-pushed. The base renamed num_workersmax_workers; this PR swaps check_function for tracking_path. Resolved as the union of both — max_workers matters because num_workers only survives on a daisy-1.x compat alias that warns.

MWE — one task, run twice, counting process_block calls:

base    run 1: 64 blocks   run 2: 64 blocks   ← silently recomputes everything
branch  run 1: 64 blocks   run 2:  0 blocks   ← resumes

Tests: 191 passed, 3 failed. The same 3 (tests/test_logging.py) fail on the base with this branch absent — pre-existing, not from this change.

Two things before it leaves draft:

  1. Add the regression test — run twice, assert zero recomputation, and assert a failed block is still retried. For a bug whose defining property was silence, no test is the wrong economy.
  2. Fix the Tests section — "3 failed, 182 passed on both main and this branch" was measured before the retarget. Use the numbers above.

📌 check_block_func now has no production caller (this PR removed its only one) but still has a test. Left alone deliberately: removing it is a different change from fixing resume.

@rhoadesScholar
rhoadesScholar changed the base branch from main to will/rbws-plus-main August 12, 2026 19:38
…resume-conflict-fix

# Conflicts:
#	volara/blockwise/blockwise.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants