Skip to content

Speed up finding the partial fanout chunk size - #2849

Open
vrom911 wants to merge 6 commits into
masterfrom
vrom911/optimise-partial-fanout
Open

Speed up finding the partial fanout chunk size#2849
vrom911 wants to merge 6 commits into
masterfrom
vrom911/optimise-partial-fanout

Conversation

@vrom911

@vrom911 vrom911 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Resolves #2848

Three things were improved with the search:

  1. It was bounded by the size of the set being distributed, so a 4000-output head searched [1..3999] — twelve candidates per step, the first ones carrying over a thousand outputs. Verifying n outputs needs n+1 CRS points, so nothing above the deployed CRS length can ever be valid.
  2. The whole-set transaction we try first was built even when far too large: a membership proof over the whole set, 141 kB of transaction at head 4000, rejected on size every step.
  3. Every candidate rebuilt the accumulator over the whole proof set — which does not depend on the chunk size — and the remaining accumulator from scratch.

Search time per fanout step, ada-only head, 3 parties:

head before after
400 302 ms 115 ms
1000 421 ms 155 ms
2000 1281 ms 229 ms
4000 1293 ms 346 ms

Candidate transactions built per step: ceil(log2 headSize) + 15, constant in head size (12 → 5 at head 4000).

Behaviour is unchanged: same chunk chosen (23 ada-only outputs at every head size), same errors, same transaction submitted. Two new tests cover it: one asserts the candidate count no longer grows with the head, the other that the chosen chunk is still maximal (it fits, one more doesn't), which is the guard against capping too tightly.


  • CHANGELOG updated or not needed
  • Documentation updated or not needed
  • Haddocks updated or not needed
  • No new TODOs introduced or explained herafter

@vrom911 vrom911 self-assigned this Aug 27, 2026
Finding the chunk size means building several candidate transactions,
and partialFanout rebuilt everything for each one: the head output, the
datum, the accumulator over the whole proof set - which does not depend
on the chunk size at all - and the remaining accumulator from scratch.

Split it into preparePartialFanout, which does the chunk-independent
work once and verifies the accumulator, and partialFanoutFromPlan, which
builds a candidate from that plan. The remaining accumulator now comes
from applyUTxODelta on the verified one instead of a full rebuild.
partialFanout stays as the composition of the two.

Search time per step on an ada-only head:

  head 400   165ms -> 115ms
  head 1000  245ms -> 155ms
  head 2000  352ms -> 229ms
  head 4000  1164ms -> 346ms

Capping the search alone had left the 4000 case untouched; this is where
that cost was. Chunk chosen is unchanged.
@vrom911
vrom911 force-pushed the vrom911/optimise-partial-fanout branch from 1d2ea9c to 080a1db Compare August 27, 2026 16:32
@github-actions

Copy link
Copy Markdown

Transaction cost differences

No cost or size differences found

@github-actions

github-actions Bot commented Aug 27, 2026

Copy link
Copy Markdown

End-to-end benchmark differences

Comparing f001c37 (PR) against merge-base 59737ae. Each runner measures both sides; every delta is the median over the same-machine pairs. Colored rows exceed the per-metric noise threshold with directional agreement (a calibrated heuristic, not a significance test); is within noise; uncolored rows are context. 🟢 = improvement, 🔴 = regression.

Round-trip latency (3 nodes, closed-loop, 3x250 txs)

Metric master PR Δ
End-to-end TPS (tx/s) 165.87 165.77 ≈ -0.1%
Sustained TPS, slope (tx/s) 168.82 166.57 ≈ -1.0%
Backlog drain time (s) 0.01 0.01 ≈ -0.1%
Snapshots per second (/s) 113.18 112.83 ≈ -0.1%
Avg txs per snapshot 1.47 1.46 ≈ -0.1%
Avg. Confirmation Time (s) 0.018 0.018 ≈ +0.4%
P50 confirmation (s) 0.017 0.017 ≈ +0.6%
P95 confirmation (s) 0.023 0.023 ≈ -0.6%
Tx validation time p50 (s) 0.005 0.005 ≈ +1.1%
Alloc MB per confirmed tx 4.13 4.13 ≈ -0.1%
Alloc MB per snapshot 6.07 6.05 ≈ -0.3%
Mutator CPU s per 1k txs 13.63 13.68 ≈ +0.5%
Max live MB (max node) 16.36 16.36 ≈ +0.0%
Peak node RSS (MB) 126.54 126.87 ≈ +0.3%
Invalid txs 0.00 0.00 ≈ +0.0

Sustained load (3 nodes, 3x5000 txs)

Metric master PR Δ
End-to-end TPS (tx/s) 711.71 710.45 ≈ -1.1%
Sustained TPS, slope (tx/s) 1,150.38 1,084.91 ≈ -6.7%
Backlog drain time (s) 20.56 20.61 ≈ +1.1%
Snapshots per second (/s) 0.76 0.77 ≈ +2.7%
Avg txs per snapshot 937.50 909.93 ≈ -2.9%
Tx validation time p50 (s) 5.489 5.195 ≈ -3.7%
Alloc MB per confirmed tx 3.80 3.84 ≈ +1.3%
Alloc MB per snapshot 3,557.97 3,497.51 ≈ -1.7%
Mutator CPU s per 1k txs 3.90 3.97 ≈ +1.6%
Max live MB (max node) 79.86 80.43 ≈ +1.1%
Peak node RSS (MB) 334.44 331.11 ≈ -0.5%
Invalid txs 0.00 0.00 ≈ +0.0

Plateau 1000 UTxO (1 node, 4000 txs)

Metric master PR Δ
End-to-end TPS (tx/s) 644.50 657.37 ≈ +1.0%
Sustained TPS, slope (tx/s) 626.18 636.12 ≈ -12.8% (3/4 pairs)
Backlog drain time (s) 6.15 6.03 ≈ -1.0%
Snapshots per second (/s) 2.11 1.71 -25.1%
Avg txs per snapshot 339.87 422.22 ≈ +45.0%
Tx validation time p50 (s) 2.657 2.602 ≈ -9.0%
Alloc MB per confirmed tx 3.07 3.09 ≈ +3.2%
Alloc MB per snapshot 1,038.51 1,341.51 +49.8%
Mutator CPU s per 1k txs 1.97 1.92 ≈ -3.8%
Max live MB (max node) 21.78 25.26 🔴 +17.4%
Peak node RSS (MB) 153.50 158.39 ≈ +3.2%
Invalid txs 0.00 0.00 ≈ +0.0

  • bench-results-m1: AMD EPYC 9V74 80-Core Processor (4 vCPU, 16 GB)
  • bench-results-m2: AMD EPYC 7763 64-Core Processor (4 vCPU, 16 GB)
  • bench-results-m3: AMD EPYC 7763 64-Core Processor (4 vCPU, 16 GB)
  • bench-results-m4: AMD EPYC 9V74 80-Core Processor (4 vCPU, 16 GB)
  • Same-code spread across runners (End-to-end TPS), the noise an unpaired comparison would see:
    • Round-trip latency (3 nodes, closed-loop, 3x250 txs): branch 27.8%, master 26.6%
    • Sustained load (3 nodes, 3x5000 txs): branch 47.9%, master 49.1%
    • Plateau 1000 UTxO (1 node, 4000 txs): branch 37.1%, master 47.9%
Per-run raw values
Machine Slot Side Scenario E2E TPS Outcome
bench-results-m1 1 branch Round-trip latency (3 nodes, closed-loop, 3x250 txs) 206.84 ok
bench-results-m1 1 branch Sustained load (3 nodes, 3x5000 txs) 973.64 ok
bench-results-m1 1 branch Plateau 1000 UTxO (1 node, 4000 txs) 826.88 ok
bench-results-m1 2 master Round-trip latency (3 nodes, closed-loop, 3x250 txs) 206.42 ok
bench-results-m1 2 master Sustained load (3 nodes, 3x5000 txs) 990.68 ok
bench-results-m1 2 master Plateau 1000 UTxO (1 node, 4000 txs) 807.44 ok
bench-results-m2 1 master Round-trip latency (3 nodes, closed-loop, 3x250 txs) 168.17 ok
bench-results-m2 1 master Sustained load (3 nodes, 3x5000 txs) 683.34 ok
bench-results-m2 1 master Plateau 1000 UTxO (1 node, 4000 txs) 545.96 ok
bench-results-m2 2 branch Round-trip latency (3 nodes, closed-loop, 3x250 txs) 167.50 ok
bench-results-m2 2 branch Sustained load (3 nodes, 3x5000 txs) 673.85 ok
bench-results-m2 2 branch Plateau 1000 UTxO (1 node, 4000 txs) 657.73 ok
bench-results-m3 1 branch Round-trip latency (3 nodes, closed-loop, 3x250 txs) 164.03 ok
bench-results-m3 1 branch Sustained load (3 nodes, 3x5000 txs) 658.35 ok
bench-results-m3 1 branch Plateau 1000 UTxO (1 node, 4000 txs) 603.10 ok
bench-results-m3 2 master Round-trip latency (3 nodes, closed-loop, 3x250 txs) 163.57 ok
bench-results-m3 2 master Sustained load (3 nodes, 3x5000 txs) 664.37 ok
bench-results-m3 2 master Plateau 1000 UTxO (1 node, 4000 txs) 629.25 ok
bench-results-m4 1 master Round-trip latency (3 nodes, closed-loop, 3x250 txs) 163.10 ok
bench-results-m4 1 master Sustained load (3 nodes, 3x5000 txs) 740.07 ok
bench-results-m4 1 master Plateau 1000 UTxO (1 node, 4000 txs) 659.76 ok
bench-results-m4 2 branch Round-trip latency (3 nodes, closed-loop, 3x250 txs) 161.79 ok
bench-results-m4 2 branch Sustained load (3 nodes, 3x5000 txs) 747.04 ok
bench-results-m4 2 branch Plateau 1000 UTxO (1 node, 4000 txs) 657.00 ok

Workflow run

@github-actions

github-actions Bot commented Aug 27, 2026

Copy link
Copy Markdown
Transaction costs

Transaction costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at 2026-08-28 15:17:08.510594617 UTC
Max. memory units 14000000
Max. CPU units 10000000000
Max. tx size (kB) 16384

Script summary

Name Hash Size (Bytes)
νHead 1d511733200df551c8cd8cddb3160ed39087af815638be37a1b80ffd 12962
μHead 603f261bc5a01f5f4ed988cc07a40a130b94e6820c28e9affde82d9e* 4971
νDeposit eafae2c32f99ab347c7bb15961e0e84c74305f9088c1a7b8abf88e7f 2117
νCRS 09db7ee6cf7a4b358dd5c8a2f19d2c048336ffc5a01ef35a47ca7072 2736
  • The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per head.

Init transaction costs

Parties Tx size % max Mem % max CPU Min fee ₳
1 5480 9.15 3.00 0.49
2 5569 10.02 3.29 0.51
3 5668 10.29 3.37 0.51
5 5862 11.30 3.69 0.53
10 6341 14.04 4.58 0.58
50 10180 35.12 11.15 0.97
100 14982 62.14 19.60 1.46
114 16330 69.30 21.86 1.59

Cost of Increment Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 2823 20.96 7.51 0.70
2 2954 22.44 8.66 0.72
3 3089 23.43 9.64 0.74
5 3346 24.75 11.38 0.78
10 4006 29.79 16.32 0.88
50 9243 73.32 56.60 1.76
75 12518 98.26 81.03 2.28

Cost of Decrement Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 642 18.52 6.70 0.58
2 773 19.49 7.67 0.60
3 902 20.46 8.65 0.62
5 1167 22.41 10.60 0.66
10 1821 27.23 15.46 0.76
50 7063 68.02 54.87 1.61
75 10338 93.24 79.43 2.13

Close transaction costs

Parties Tx size % max Mem % max CPU Min fee ₳
10 1849 26.50 20.46 0.79
75 10369 92.81 84.66 2.17

Contest transaction costs

Parties Tx size % max Mem % max CPU Min fee ₳
1 705 21.47 14.90 0.66
2 828 22.65 15.94 0.68
3 963 23.78 16.96 0.71
5 1230 26.04 19.01 0.75
10 1876 31.80 24.17 0.86
50 7122 79.56 65.74 1.78
66 9217 99.55 82.61 2.16

FanOut transaction costs

Involves spending head output and burning head tokens. Uses ada-only UTXO for better comparability.
Rows first grow the UTxO set at a fixed 10 parties, then show the largest set that still fits per number of parties (burning more participation tokens leaves less room for outputs).

Parties UTxO UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
10 0 0 5644 23.70 43.00 1.14
10 1 57 5678 26.05 45.51 1.18
10 5 283 5812 36.28 55.83 1.34
10 10 570 5984 50.26 69.12 1.56
10 20 1138 6323 82.76 97.10 2.04
1 20 1137 6042 76.54 95.13 1.96
5 20 1136 6165 79.31 96.00 1.99
10 20 1140 6325 82.76 97.10 2.04
20 20 1140 6634 90.08 99.39 2.13
50 15 853 7393 95.05 92.01 2.15

PartialFanOut transaction costs

Largest chunk of ada-only outputs that can be distributed in one partial fanout step, computed dynamically. The last row is the maximum total UTxO count where at least one output can still be distributed.

Total UTxO Distributed UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
11 10 569 986 35.29 66.46 1.20
25 23 1310 1429 68.58 99.53 1.73
30 23 1307 1426 68.58 99.53 1.73
40 23 1308 1427 68.58 99.53 1.73
50 23 1309 1428 68.58 99.53 1.73
100 23 1310 1429 68.58 99.53 1.73
150 23 1311 1430 68.58 99.53 1.73
200 23 1310 1429 68.58 99.53 1.73
200 23 1309 1428 68.58 99.53 1.73

PartialFanOut transaction costs (with native tokens)

Largest chunk of native-token outputs that can be distributed in one partial fanout step, computed dynamically. The last row is the maximum total UTxO count where at least one output can still be distributed.

Total UTxO Distributed UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
11 10 1090 1576 42.25 68.95 1.30
25 21 2268 2522 76.69 99.25 1.84
30 21 1974 2214 76.69 99.19 1.82
40 21 2247 2500 76.71 99.25 1.84
50 21 2415 2677 76.71 99.30 1.84
100 21 2079 2325 76.71 99.20 1.83
150 21 2037 2281 76.71 99.20 1.83
200 21 2541 2809 76.71 99.35 1.85
200 21 2163 2413 76.69 99.24 1.83

FinalPartialFanOut transaction costs (with native tokens)

Terminal partial fanout step (FanoutProgress → Final) with outputs carrying a native token. Burns all head tokens and proves accumulator exhaustion via BLS proof.

Distributed UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
1 109 5528 22.39 44.40 1.14
5 540 5876 35.93 55.92 1.34
10 940 6171 54.06 70.67 1.61
10 1050 6280 54.18 70.73 1.61
End-to-end benchmark results

End-to-end benchmark results

This page is intended to collect the latest end-to-end benchmark results produced by Hydra's continuous integration (CI) system from the latest master code.

Please note that these results are approximate as they are currently produced from limited cloud VMs and not controlled hardware. Rather than focusing on the absolute results, the emphasis should be on relative results, such as how the timings for a scenario evolve as the code changes.

Generated at 2026-08-28 15:19:21.111207327 UTC

Baseline Scenario

Number of nodes 1
Number of txs 300
Load mode open-loop
Avg. Confirmation Time (ms) 198.2
P99 201.9ms
P95 201.5ms
P50 198.1ms
Tx validation time p50 (ms) 142.1
End-to-end TPS 1481.41 tx/s
Backlog drain time (s) 0.2
Snapshots observed 2
Snapshots per second 9.88 /s
Avg txs per snapshot 150.0
Peak node RSS (MB) 129.1
Number of Invalid txs 0
Fanout outputs 2

Three local nodes

Number of nodes 3
Number of txs 900
Load mode open-loop
Avg. Confirmation Time (ms) 901.5
P99 955.1ms
P95 954.0ms
P50 902.7ms
Tx validation time p50 (ms) 495.8
End-to-end TPS 931.55 tx/s
Backlog drain time (s) 0.9
Snapshots observed 3
Snapshots per second 3.11 /s
Avg txs per snapshot 300.0
Peak node RSS (MB) 145.0
Number of Invalid txs 0
Fanout outputs 4
Scenario benchmark results

Scenario benchmark results

This page collects results from the scenario matrix: every combination of cluster size, UTxO shape, and incremental-ops mode is exercised by CI from the latest master code and reported below.

Numbers are approximate. They come from cloud VMs rather than controlled hardware, so the useful signal is the relative change between cells and between commits, not the absolute throughput.

Generated at 2026-08-28 15:32:20.239067306 UTC

Summary across cells

TPS columns are rates (transactions per second); Wall clock (s) is the measured elapsed time from the first tx submission to the last confirmation. Times are rounded to one decimal.

Scenario Txs Wall clock (s) End-to-end TPS (tx/s) Sustained TPS (tx/s) Avg conf (ms) P95 conf (ms)
Nodes=1, Constant, fire and forget 30 0.0 1297.96 n/a 22.4 22.9
Nodes=1, Constant, wait for tx valid 30 0.1 216.15 215.51 4.6 5.3
Nodes=1, Growing, fire and forget 30 0.0 888.94 n/a 33.0 33.5
Nodes=1, Growing, wait for tx valid 30 0.2 170.78 169.93 5.8 7.3
Nodes=1, Mixed, fire and forget 30 0.0 891.78 n/a 32.8 33.4
Nodes=1, Mixed, wait for tx valid 30 0.2 191.73 190.51 5.1 5.9
Nodes=2, Constant, fire and forget 60 0.1 1083.96 n/a 54.0 54.4
Nodes=2, Constant, wait for tx valid 60 0.4 146.99 146.55 13.4 17.5
Nodes=2, Growing, fire and forget 60 0.1 985.89 n/a 59.1 59.9
Nodes=2, Growing, wait for tx valid 60 0.5 110.85 108.51 17.8 21.0
Nodes=2, Mixed, fire and forget 60 0.1 540.09 n/a 110.0 110.5
Nodes=2, Mixed, wait for tx valid 60 0.5 112.81 107.62 17.4 22.7
Nodes=3, Constant, fire and forget 90 0.1 729.56 n/a 121.5 122.7
Nodes=3, Constant, wait for tx valid 90 0.8 118.26 114.02 25.1 34.4
Nodes=3, Growing, fire and forget 90 0.1 668.72 n/a 130.3 134.3
Nodes=3, Growing, wait for tx valid 90 1.1 85.64 84.24 34.7 42.2
Nodes=3, Mixed, fire and forget 90 0.1 656.15 n/a 133.9 136.8
Nodes=3, Mixed, wait for tx valid 90 1.0 93.42 91.42 31.5 39.1

Nodes=1, Constant, fire and forget

Number of nodes 1
Number of txs 30
Load mode open-loop
Avg. Confirmation Time (ms) 22.4
P99 22.9ms
P95 22.9ms
P50 22.5ms
Tx validation time p50 (ms) 9.6
End-to-end TPS 1297.96 tx/s
Backlog drain time (s) 0.0
Snapshots observed 2
Snapshots per second 86.53 /s
Avg txs per snapshot 15.0
Peak node RSS (MB) 142.6
Number of Invalid txs 0
Fanout outputs 2

Nodes=1, Constant, wait for tx valid

Number of nodes 1
Number of txs 30
Load mode closed-loop
Avg. Confirmation Time (ms) 4.6
P99 7.0ms
P95 5.3ms
P50 4.3ms
Tx validation time p50 (ms) 1.6
End-to-end TPS 216.15 tx/s
Sustained TPS 215.51 tx/s
Backlog drain time (s) 0.0
Snapshots observed 30
Snapshots per second 216.15 /s
Avg txs per snapshot 1.0
Peak node RSS (MB) 142.4
Number of Invalid txs 0
Fanout outputs 2

Nodes=1, Growing, fire and forget

Number of nodes 1
Number of txs 30
Load mode open-loop
Avg. Confirmation Time (ms) 33.0
P99 33.5ms
P95 33.5ms
P50 33.2ms
Tx validation time p50 (ms) 17.4
End-to-end TPS 888.94 tx/s
Backlog drain time (s) 0.0
Snapshots observed 2
Snapshots per second 59.26 /s
Avg txs per snapshot 15.0
Peak node RSS (MB) 144.4
Number of Invalid txs 0
Fanout outputs 31

Nodes=1, Growing, wait for tx valid

Number of nodes 1
Number of txs 30
Load mode closed-loop
Avg. Confirmation Time (ms) 5.8
P99 7.8ms
P95 7.3ms
P50 5.6ms
Tx validation time p50 (ms) 1.6
End-to-end TPS 170.78 tx/s
Sustained TPS 169.93 tx/s
Backlog drain time (s) 0.0
Snapshots observed 30
Snapshots per second 170.78 /s
Avg txs per snapshot 1.0
Peak node RSS (MB) 130.5
Number of Invalid txs 0
Fanout outputs 31

Nodes=1, Mixed, fire and forget

Each client first grows its UTxO set (1-in to 2-out) for half of its tx budget, then contracts it back (2-in to 1-out) for the remainder.

Number of nodes 1
Number of txs 30
Load mode open-loop
Avg. Confirmation Time (ms) 32.8
P99 33.4ms
P95 33.4ms
P50 33.0ms
Tx validation time p50 (ms) 14.5
End-to-end TPS 891.78 tx/s
Backlog drain time (s) 0.0
Snapshots observed 2
Snapshots per second 59.45 /s
Avg txs per snapshot 15.0
Peak node RSS (MB) 144.8
Number of Invalid txs 0
Fanout outputs 2

Nodes=1, Mixed, wait for tx valid

Each client first grows its UTxO set (1-in to 2-out) for half of its tx budget, then contracts it back (2-in to 1-out) for the remainder.

Number of nodes 1
Number of txs 30
Load mode closed-loop
Avg. Confirmation Time (ms) 5.1
P99 7.1ms
P95 5.9ms
P50 5.0ms
Tx validation time p50 (ms) 1.6
End-to-end TPS 191.73 tx/s
Sustained TPS 190.51 tx/s
Backlog drain time (s) 0.0
Snapshots observed 30
Snapshots per second 191.73 /s
Avg txs per snapshot 1.0
Peak node RSS (MB) 128.7
Number of Invalid txs 0
Fanout outputs 2

Nodes=2, Constant, fire and forget

Number of nodes 2
Number of txs 60
Load mode open-loop
Avg. Confirmation Time (ms) 54.0
P99 54.5ms
P95 54.4ms
P50 54.1ms
Tx validation time p50 (ms) 25.6
End-to-end TPS 1083.96 tx/s
Backlog drain time (s) 0.1
Snapshots observed 2
Snapshots per second 36.13 /s
Avg txs per snapshot 30.0
Peak node RSS (MB) 144.6
Number of Invalid txs 0
Fanout outputs 3

Nodes=2, Constant, wait for tx valid

Number of nodes 2
Number of txs 60
Load mode closed-loop
Avg. Confirmation Time (ms) 13.4
P99 20.5ms
P95 17.5ms
P50 12.7ms
Tx validation time p50 (ms) 3.6
End-to-end TPS 146.99 tx/s
Sustained TPS 146.55 tx/s
Backlog drain time (s) 0.0
Snapshots observed 60
Snapshots per second 146.99 /s
Avg txs per snapshot 1.0
Peak node RSS (MB) 144.1
Number of Invalid txs 0
Fanout outputs 3

Nodes=2, Growing, fire and forget

Number of nodes 2
Number of txs 60
Load mode open-loop
Avg. Confirmation Time (ms) 59.1
P99 60.0ms
P95 59.9ms
P50 59.5ms
Tx validation time p50 (ms) 32.6
End-to-end TPS 985.89 tx/s
Backlog drain time (s) 0.1
Snapshots observed 2
Snapshots per second 32.86 /s
Avg txs per snapshot 30.0
Peak node RSS (MB) 144.7
Number of Invalid txs 0
Fanout outputs 62

Nodes=2, Growing, wait for tx valid

Number of nodes 2
Number of txs 60
Load mode closed-loop
Avg. Confirmation Time (ms) 17.8
P99 23.1ms
P95 21.0ms
P50 18.0ms
Tx validation time p50 (ms) 5.9
End-to-end TPS 110.85 tx/s
Sustained TPS 108.51 tx/s
Backlog drain time (s) 0.0
Snapshots observed 60
Snapshots per second 110.85 /s
Avg txs per snapshot 1.0
Peak node RSS (MB) 145.7
Number of Invalid txs 0
Fanout outputs 62

Nodes=2, Mixed, fire and forget

Each client first grows its UTxO set (1-in to 2-out) for half of its tx budget, then contracts it back (2-in to 1-out) for the remainder.

Number of nodes 2
Number of txs 60
Load mode open-loop
Avg. Confirmation Time (ms) 110.0
P99 110.5ms
P95 110.5ms
P50 110.2ms
Tx validation time p50 (ms) 75.9
End-to-end TPS 540.09 tx/s
Backlog drain time (s) 0.1
Snapshots observed 2
Snapshots per second 18.00 /s
Avg txs per snapshot 30.0
Peak node RSS (MB) 144.5
Number of Invalid txs 0
Fanout outputs 3

Nodes=2, Mixed, wait for tx valid

Each client first grows its UTxO set (1-in to 2-out) for half of its tx budget, then contracts it back (2-in to 1-out) for the remainder.

Number of nodes 2
Number of txs 60
Load mode closed-loop
Avg. Confirmation Time (ms) 17.4
P99 24.5ms
P95 22.7ms
P50 17.7ms
Tx validation time p50 (ms) 5.0
End-to-end TPS 112.81 tx/s
Sustained TPS 107.62 tx/s
Backlog drain time (s) 0.0
Snapshots observed 60
Snapshots per second 112.81 /s
Avg txs per snapshot 1.0
Peak node RSS (MB) 144.4
Number of Invalid txs 0
Fanout outputs 3

Nodes=3, Constant, fire and forget

Number of nodes 3
Number of txs 90
Load mode open-loop
Avg. Confirmation Time (ms) 121.5
P99 122.8ms
P95 122.7ms
P50 122.4ms
Tx validation time p50 (ms) 44.4
End-to-end TPS 729.56 tx/s
Backlog drain time (s) 0.1
Snapshots observed 2
Snapshots per second 16.21 /s
Avg txs per snapshot 45.0
Peak node RSS (MB) 144.2
Number of Invalid txs 0
Fanout outputs 4

Nodes=3, Constant, wait for tx valid

Number of nodes 3
Number of txs 90
Load mode closed-loop
Avg. Confirmation Time (ms) 25.1
P99 36.9ms
P95 34.4ms
P50 24.0ms
Tx validation time p50 (ms) 6.7
End-to-end TPS 118.26 tx/s
Sustained TPS 114.02 tx/s
Backlog drain time (s) 0.0
Snapshots observed 61
Snapshots per second 80.15 /s
Avg txs per snapshot 1.5
Peak node RSS (MB) 145.2
Number of Invalid txs 0
Fanout outputs 4

Nodes=3, Growing, fire and forget

Number of nodes 3
Number of txs 90
Load mode open-loop
Avg. Confirmation Time (ms) 130.3
P99 134.3ms
P95 134.3ms
P50 132.5ms
Tx validation time p50 (ms) 67.3
End-to-end TPS 668.72 tx/s
Backlog drain time (s) 0.1
Snapshots observed 2
Snapshots per second 14.86 /s
Avg txs per snapshot 45.0
Peak node RSS (MB) 145.2
Number of Invalid txs 0
Fanout outputs 0

Nodes=3, Growing, wait for tx valid

Number of nodes 3
Number of txs 90
Load mode closed-loop
Avg. Confirmation Time (ms) 34.7
P99 54.5ms
P95 42.2ms
P50 34.7ms
Tx validation time p50 (ms) 10.4
End-to-end TPS 85.64 tx/s
Sustained TPS 84.24 tx/s
Backlog drain time (s) 0.0
Snapshots observed 61
Snapshots per second 58.04 /s
Avg txs per snapshot 1.5
Peak node RSS (MB) 146.1
Number of Invalid txs 0
Fanout outputs 0

Nodes=3, Mixed, fire and forget

Each client first grows its UTxO set (1-in to 2-out) for half of its tx budget, then contracts it back (2-in to 1-out) for the remainder.

Number of nodes 3
Number of txs 90
Load mode open-loop
Avg. Confirmation Time (ms) 133.9
P99 136.9ms
P95 136.8ms
P50 134.5ms
Tx validation time p50 (ms) 49.1
End-to-end TPS 656.15 tx/s
Backlog drain time (s) 0.1
Snapshots observed 2
Snapshots per second 14.58 /s
Avg txs per snapshot 45.0
Peak node RSS (MB) 144.9
Number of Invalid txs 0
Fanout outputs 4

Nodes=3, Mixed, wait for tx valid

Each client first grows its UTxO set (1-in to 2-out) for half of its tx budget, then contracts it back (2-in to 1-out) for the remainder.

Number of nodes 3
Number of txs 90
Load mode closed-loop
Avg. Confirmation Time (ms) 31.5
P99 42.9ms
P95 39.1ms
P50 30.9ms
Tx validation time p50 (ms) 9.2
End-to-end TPS 93.42 tx/s
Sustained TPS 91.42 tx/s
Backlog drain time (s) 0.0
Snapshots observed 61
Snapshots per second 63.32 /s
Avg txs per snapshot 1.5
Peak node RSS (MB) 145.7
Number of Invalid txs 0
Fanout outputs 4

@vrom911
vrom911 requested a review from a team August 27, 2026 18:56
@vrom911 vrom911 moved this to In review 👀 in ☕ Hydra Team Work Aug 28, 2026
@noonio

noonio commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Edited after 614c2798d. The first version of this comment made four claims that turned out to be wrong, all of them mine: that the fanoutTx count fix was sufficient (it was not, see below), that the whole-set transaction is "only attempted when it could be valid", that a step builds at most five candidates, and the 1.3s to 0.35s timing. Corrected in place rather than left standing; 614c2798d fixes the code and prose behind them.

Follow-up commits 7ca075d and 614c2798d address a review pass over this branch. Grouped by what they change and why.

Correctness

partialFanoutFromPlan could commit to the wrong remaining set. Swapping buildFromUTxO (rest <> presettled) for applyUTxODelta fullAccumulator proofUTxO rest is not a refactor: applyUTxODelta derives its add/remove sets from the TxIn-keyed withoutUTxO, while accumulator elements are keyed by TxOut content. Hydra.HeadLogic.isSubMultisetOf validates a client selection by content and explicitly ignores TxIns, so a selection can carry an output under a TxIn that holds a different one in the set the accumulator was built from. The delta then removes the wrong element, the on-chain split identity fails, and every probed chunk fails script evaluation, so a selection that fanned out before this branch throws FailedToConstructPartialFanoutTx.

Fixed with a new Accumulator.removeOutputs, which removes the distributed outputs from the verified accumulator at element level. That is exactly what A = P_K * A' checks, so it holds by construction and does not depend on which TxIn holds an output. proofUTxO and presettled fall out of PartialFanoutPlan as a result: pre-settled elements are already inside the verified accumulator and stay there, because a step only removes what it distributes.

Two StateSpec properties fail on the pre-fix tree and pass after, the second with a Phase-2 validation failure:

  • selection holding outputs under other TxIns: accumulator still rebuilds
  • selection holding outputs under other TxIns: batch tx evaluates on-chain

Empty search range turned a terminal error into a silent no-op. preparePartialFanout ran before the search range was checked, so a head with one UTxO left whose preferred tx did not fit would report StalePartialFanoutTx when the chain state was stale, which HeadLogic ignores. findFallback now short-circuits when min maxChunkSize deployedFanoutBatchSize < 1. Test throws FailedToConstructPartialFanoutTx when no chunk size is in range fails on the pre-fix tree with exactly StalePartialFanoutTx. Note the revert this restores only fires while nothing has been distributed, so on the FinalPartialFanoutTx path the effect is the client error alone.

fanoutTx emitted more outputs than it proved. numberOfFanoutOutputs was a sum over the snapshot UTxO, pending commit and pending decommit, while the proof and the head's accumulator are built over their union. Those disagree exactly when two of the sets share a TxIn, and the transaction then fails with FanoutUTxOHashMismatch (H39).

Taking the count and the proof from one allToFanout binding is not on its own enough, which the first version of this comment got wrong. Outputs are emitted as three ascending groups and the validator checks its proof against the first numberOfFanoutOutputs of them, so overlap makes that prefix reach into a later group and it fails whenever a consumed entry is one an earlier group already emitted. Whether that happens depends on sort order, and the original regression test passed only because its fixture's shared entry happened to fall outside the prefix. Each group now drops what the groups before it emitted, so the list is a permutation of the union for any input. The fixture is now a discriminating one (shared entry sorting before the second group's own entry), verified H39 before and green after.

Not reachable through the node, for the record: the three sets are pairwise TxIn-disjoint at the one place snapshots are built. onOpenNetworkReqSn removes the commit set from the active UTxO (nextUTxO = u \withoutUTxO` mUtxoToCommit) and the decommit outputs from the confirmed one (activeUTxO = newConfirmedUTxO `withoutUTxO` utxoToDecommit), and ReqSnBothCommitAndDecommitpluseffectiveCommitDecommitmean at most one of commit/decommit ever reachesfanoutTx`.

Test correctness

  • withinSizeLimits used < where Wallet.hs:186 uses <=, so the oracle was one output tighter than the code it checked.
  • does not scale the number of candidate transactions with the head size asserted equality of two probe counts that legitimately differ by one. Now a bound on each, with non-degeneracy pinned by the maximality test.
  • Both large-head tests drew their fixture with generate inside an it, so a CI failure printed no replayable seed. Now withMaxSuccess 1 $ forAllBlind ... monadicIO.
  • picks the largest chunk that fits re-derived the chosen transaction through the same code path, so half of it could not fail. It now checks the transaction the search returned, reports which limit chunk + 1 runs into, and treats a construction failure as a failure rather than as evidence that one more does not fit.
  • propRemainingAccumulatorRebuilds's oracle was itself TxIn-keyed and would have rejected correct code for a selection naming an output under a TxIn the proof set does not hold. Now content-keyed via removeDistributedOutputs.
  • The deployedFanoutBatchSize pin restated defaultItems - 1 from the same module. It now runs checkMembershipPairing against canonicalG2Points, accepting 29 scalars and rejecting 30.

Test coverage

  • New describe "preferredFanoutTx": the laziness pin (a bottom in the Right, since never forcing the argument is the entire benefit) and the fencepost at the cap.
  • The accumulator substitution is now pinned at its call site, plus a variant with a non-empty pre-settled set, and three removeOutputs properties in AccumulatorSpec.

Efficiency and simplification

  • maxVerifiableChunk moved out of the chain-handler layer to KZGTrustedSetup.deployedFanoutBatchSize, beside defaultItems and maxFanoutBatchSize, re-exported from Hydra.Tx.Accumulator the way maxAccumulatorSize already is.
  • The tx-cost benchmark bounds its search by that constant. The testlib's findFittingPartialChunk does too, though no current caller reaches heads that large, so there it is insurance rather than a live optimisation.
  • finalPartialFanoutTx takes the verified accumulator instead of presettledUTxO, so finalPartialFanout no longer builds the same accumulator twice. Worth noting this demotes an invariant the signature used to enforce into a haddock claim.
  • The FanoutTx call site counts the preferred transaction's outputs the way fanoutTx counts them.

removeOutputs also cuts per-candidate work: the old delta rebuilt an (n - chunk)-entry map and ran two full Map.difference passes per candidate. What remains is one G1 commitment MSM per candidate, inherent since each commits to a different remaining set.

Documentation

CHANGELOG entry corrected, architecture doc's Fanout bullet and chunk-search paragraph updated, and the Caveats remedy fixed in three places: raising the cap needs the validator recompiled, not just the CRS output re-published, because the validator is compiled against hashCRSDatum canonicalG2Points.

Measurements

Timing the fallback search on a generated 4000-output ada-only head, replacing the 1.3s to 0.35s the entry used to claim:

candidate builds elapsed
uncapped range 12 1.069s
capped at 29 5 0.885s

Most of the cost is the one full-width accumulator build and MSM that both variants pay, so the win is in builds rather than wall clock. Single runs on one machine, not a benchmark harness. The chunk is 23, and the tx-cost benchmark still reports the same chunk sizes as before (23 ada-only, 21 with native tokens, flat across totals 25 to 200), so capping its search changed what it costs to produce, not what it says.

Two things worth a second opinion, neither addressed here

A reachable head wedge, pre-existing. onPartialFanoutClientPartialFanout has no sameOutputs remainingOutputs guard, unlike onClosedClientPartialFanout. A client can therefore select the whole remainder while distributedOutputs is still empty. With a non-empty pre-settled set the resulting A' is the accumulator over presettled, not the G1 generator, so mustNotBeLastBatch passes and the transaction is valid; remaining is then empty, the next step is a zero-output FinalPartialFanoutTx that cannot be built, and the revert guard at HeadLogic.hs:2243 requires nullOutputs distributedOutputs, which is now false. The head stays in FanoutProgress with tokens unburnable and headAdaOverhead locked. master has the same bound, and the new cap narrows exposure to remainders of 29 or fewer. Probably wants its own issue and its own fix.

TxTraceSpec's guard-based assertions are vacuous, pre-existing. PostconditionM' is ExceptT (Maybe String), so guard False throws Nothing and runPostconditionM' maps Left Nothing to pass. That silently disables the Fanout output comparison and the Contest contester-count check.

@noonio
noonio force-pushed the vrom911/optimise-partial-fanout branch from 614c279 to f001c37 Compare August 28, 2026 15:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In review 👀

Development

Successfully merging this pull request may close these issues.

Partial fanout does redundant work when searching for the chunk size

2 participants