Speed up finding the partial fanout chunk size

CHANGELOG.md

@@ -10,6 +10,41 @@ changes.
 
 ## [UNRELEASED]
 
+- Speed up posting a partial fanout step: the chunk size search was bounded by
+  the size of the set being distributed, so a 4000-output head built twelve
+  candidate transactions per step, the first of them carrying over a thousand
+  outputs, and rebuilt the head's accumulators for every one of them. The
+  search is now bounded by the largest subset the head validator's CRS can
+  verify, the whole-set transaction is skipped outright when it is above that
+  bound and so could never validate, and the head's accumulator is verified once
+  for the whole search rather than once per candidate. That is at most five
+  fallback candidates per step regardless of head size (six including the
+  whole-set transaction, where that is still attempted), against a previous
+  bound of `floor(log2 (n - 1)) + 1` in the size `n` of the set being
+  distributed. Measured on a 4000-output head, the fallback search builds five
+  candidates in about 0.3s where it used to build twelve in about 1s. The
+  chunk chosen is unchanged. Three limits bound a step, and the chunk is the
+  smallest of them: the layer 1 transaction size, the script execution budget,
+  and the CRS. Today the budget is the one that binds first, at 23 ada-only
+  outputs or 21 carrying native tokens, under a CRS ceiling of 29 that no step
+  can exceed whatever the other two allow. That
+  ceiling now lives beside the trusted setup it derives from, as
+  `KZGTrustedSetup.deployedFanoutBatchSize`, so the chain layer, the
+  transaction-cost benchmark and the test generators all bound their searches by
+  the same value.
+  [#2848](https://github.com/cardano-scaling/hydra/issues/2848)
+
+- Fixed `fanoutTx` counting the outputs it claims differently from the set it
+  proves membership for: the `numberOfFanoutOutputs` redeemer was a sum over the
+  snapshot UTxO, pending commit and pending decommit, while the proof and the
+  head's accumulator are built over their union. Those disagree exactly when two
+  of the sets share a `TxIn`, and the transaction then failed on chain with
+  `FanoutUTxOHashMismatch`. Both now come from the union, and each group of
+  outputs drops what the groups before it already emitted, so the emitted list is
+  a permutation of that union whatever the inputs. Snapshot construction keeps
+  the three sets disjoint, so this was not reachable through the node.
+  [#2848](https://github.com/cardano-scaling/hydra/issues/2848)
+
 - Fixed the internal wallet setting a script integrity hash on transactions
   that execute no scripts: reference inputs carrying Plutus scripts had their
   language views hashed even when nothing runs, so the ledger rejected such

docs/docs/dev/architecture/partial-fanout.md

@@ -8,9 +8,11 @@ until the head is empty.
 
 There are two ways to drive it:
 
-- a plain `Fanout` builds the single full transaction and checks whether it would
-  fit; when it would not, the head is drained automatically over as many steps as
-  it takes
+- a plain `Fanout` distributes the whole set in one transaction where that can be
+  valid at all, and otherwise drains the head automatically over as many steps as
+  it takes. A head holding more outputs than the deployed CRS can verify (see
+  [How much fits in one step](#how-much-fits-in-one-step)) takes the second path
+  without building the single transaction, which could only be rejected
 - a `PartialFanout` names the subset to distribute next, so a client decides what
   leaves the head first and in which order
 
@@ -162,9 +164,12 @@ needs to know. The chain layer runs a binary search
 ([`findFittingFanoutTx`](pathname:///haddocks/hydra-node/Hydra-Chain-Direct-Handlers.html#v:findFittingFanoutTx))
 for the largest number of outputs whose transaction still fits the layer 1 size
 limit and script execution budget, so a large selection is drained over as many
-steps as needed. The search is local: candidate transactions are built and
-evaluated against those limits in the node, and only the winning one is submitted.
-Nothing is submitted speculatively and rejected by the chain.
+steps as needed. The search never looks above what the deployed CRS can verify,
+since no larger subset could be valid however cheap its transaction turns out to
+be, which is what keeps the number of candidates independent of how much is left
+in the head. The search is local: candidate transactions are built and evaluated
+against those limits in the node, and only the winning one is submitted. Nothing
+is submitted speculatively and rejected by the chain.
 
 The last transaction cannot be an ordinary partial step: it must be the _final_
 fanout, which distributes the rest and burns the head tokens. The node handles
@@ -203,9 +208,10 @@ Two separate limits bound a single fanout transaction:
   benchmarks for the measured chunk sizes
 - the length of the deployed CRS. Verifying a subset of `N` outputs needs `N + 1`
   G2 points, and the CRS output currently carries 30 of them (`defaultItems`), so
-  no step can distribute more than **29** outputs regardless of budget. Raising
-  this means re-publishing the CRS output with more points, up to the 65 G2 points
-  the trusted setup provides
+  no step can distribute more than **29** outputs regardless of budget. The head
+  validator is compiled against the hash of that CRS and rejects any other one, so
+  raising this means re-publishing the CRS output with more points _and_
+  recompiling the scripts, up to the 65 G2 points the trusted setup provides
 
 ### UTxO sizing
 

hydra-node/bench/tx-cost/TxCost.hs

@@ -34,6 +34,7 @@ import Hydra.Ledger.Cardano.Evaluate (
 import Hydra.Ledger.Cardano.Time (slotNoFromUTCTime)
 import Hydra.Plutus.Orphans ()
 import Hydra.Tx (utxoFromTx)
+import Hydra.Tx.Accumulator (deployedFanoutBatchSize)
 import PlutusLedgerApi.V3 (toBuiltinData)
 import PlutusTx.Builtins (lengthOfByteString, serialiseData)
 import Test.Hydra.Chain.Direct.State (
@@ -208,8 +209,9 @@ computeFanOutCost = do
 -- | Compute costs of partial fanout transactions across a range of per-step
 -- distribution sizes.
 --
--- For each total UTxO count N, we build one partial fanout that distributes
--- all-but-one outputs (the benchmark shows max-chunk scaling). The tx size
+-- For each total UTxO count N, we search for the largest chunk that fits, the
+-- same way the node does: bounded by 'deployedFanoutBatchSize', since no larger
+-- subset can be verified on chain however cheap its transaction is. The tx size
 -- grows with the total UTxO count because the accumulator serialisation stored
 -- in the output datum grows linearly with remaining UTxO count.
 computePartialFanOutNominalCost :: Gen [(NumUTxO, NumUTxO, Natural, TxSize, MemUnit, CpuUnit, Coin)]
@@ -251,7 +253,7 @@ computePartialFanOutNominalCost = do
            in fmap
                 (\(txSize, memUnit, cpuUnit, minFee) -> (NumUTxO totalUTxO, NumUTxO (totalUTxO - n), serializedSize utxoDistributed, txSize, memUnit, cpuUnit, minFee))
                 (checkSizeAndEvaluate tx spendableUTxO)
-    either (const Nothing) Just <$> findLargestFitting (pure . tryChunk) (totalUTxO - 1)
+    either (const Nothing) Just <$> findLargestFitting (pure . tryChunk) (min (totalUTxO - 1) deployedFanoutBatchSize)
 
 -- | Like 'computePartialFanOutNominalCost' but uses outputs carrying native
 -- tokens (all sharing one policy ID so the accumulated head value stays
@@ -292,7 +294,7 @@ computePartialFanOutMixedCost = do
            in fmap
                 (\(txSize, memUnit, cpuUnit, minFee) -> (NumUTxO totalUTxO, NumUTxO (totalUTxO - n), serializedSize utxoDistributed, txSize, memUnit, cpuUnit, minFee))
                 (checkSizeAndEvaluate tx spendableUTxO)
-    either (const Nothing) Just <$> findLargestFitting (pure . tryChunk) (totalUTxO - 1)
+    either (const Nothing) Just <$> findLargestFitting (pure . tryChunk) (min (totalUTxO - 1) deployedFanoutBatchSize)
 
 -- | Compute costs of the final partial fanout transaction (FanoutProgress → Final)
 -- with mixed UTxOs. This is the terminal step that burns all head tokens and

hydra-node/src/Hydra/Chain/Direct/Handlers.hs

@@ -80,7 +80,8 @@ import Hydra.Chain.Direct.State (
   getKnownUTxO,
   increment,
   initialize,
-  partialFanout,
+  partialFanoutFromPlan,
+  preparePartialFanout,
   recover,
  )
 import Hydra.Chain.Direct.TimeHandle (TimeHandle (..))
@@ -102,6 +103,7 @@ import Hydra.Tx (
   UTxOType,
   headSeedToTxIn,
  )
+import Hydra.Tx.Accumulator qualified as Accumulator
 import Hydra.Tx.ContestationPeriod (toNominalDiffTime)
 import Hydra.Tx.Deposit (DepositObservation (..), depositTx)
 import Hydra.Tx.Observe (
@@ -203,14 +205,17 @@ mkChain tracer queryTimeHandle wallet ctx LocalChainState{getLatest} submitTx =
         vtx <- case tx of
           FanoutTx{utxo, utxoToCommit, utxoToDecommit, utxoForProof, headSeed, contestationDeadline} -> do
             (deadlineSlot, seedTxIn) <- resolveHeadInfo headSeed contestationDeadline
+            -- The union is also what 'Hydra.Tx.Fanout.fanoutTx' counts in its
+            -- redeemer and proves membership for, so its size is the exact
+            -- gate for 'preferredFanoutTx'.
             let fullUTxO = utxo <> fold utxoToCommit <> fold utxoToDecommit
             findFittingFanoutTx
               tracer
               wallet
               ctx
               spendableUTxO
               seedTxIn
-              (rightToMaybe (fanout ctx spendableUTxO seedTxIn utxo utxoToCommit utxoToDecommit utxoForProof deadlineSlot))
+              (preferredFanoutTx (UTxO.size fullUTxO) (fanout ctx spendableUTxO seedTxIn utxo utxoToCommit utxoToDecommit utxoForProof deadlineSlot))
               utxoForProof
               fullUTxO
               (UTxO.size fullUTxO - 1)
@@ -224,7 +229,7 @@ mkChain tracer queryTimeHandle wallet ctx LocalChainState{getLatest} submitTx =
               ctx
               spendableUTxO
               seedTxIn
-              (rightToMaybe (finalPartialFanout ctx spendableUTxO seedTxIn utxoToDistribute presettledUTxO deadlineSlot))
+              (preferredFanoutTx (UTxO.size utxoToDistribute) (finalPartialFanout ctx spendableUTxO seedTxIn utxoToDistribute presettledUTxO deadlineSlot))
               (utxoToDistribute <> presettledUTxO)
               utxoToDistribute
               (UTxO.size utxoToDistribute - 1)
@@ -234,11 +239,25 @@ mkChain tracer queryTimeHandle wallet ctx LocalChainState{getLatest} submitTx =
             (deadlineSlot, seedTxIn) <- resolveHeadInfo headSeed contestationDeadline
             -- Non-final partial fanout: no preferred tx, always chunk from the
             -- user-selected set. The whole selection may be distributed in one
-            -- tx (size, not size-1): the selection is always a strict subset of
-            -- the head's remaining UTxO (HeadLogic routes a full selection to
-            -- the final/auto path instead), so the unselected remainder stays in
-            -- the accumulator and 'mustNotBeLastBatch' is satisfied regardless of
-            -- chunk size.
+            -- tx (size, not size-1): normally the selection is a strict subset
+            -- of the head's remaining UTxO, so the unselected remainder stays in
+            -- the accumulator and 'mustNotBeLastBatch' is satisfied regardless
+            -- of chunk size.
+            --
+            -- Only the first selection is checked against that:
+            -- 'Hydra.HeadLogic.onClosedClientPartialFanout' routes a full one to
+            -- the auto-drain path, but 'onPartialFanoutClientPartialFanout' has
+            -- no such guard, so a later selection naming the whole remainder
+            -- before any chunk has landed reaches here as a full set. Candidates
+            -- are evaluated locally and only the winner is submitted, so nothing
+            -- is rejected on chain either way. What happens next depends on the
+            -- head: with nothing pre-settled the top candidate leaves an empty
+            -- accumulator, fails 'mustNotBeLastBatch' under local evaluation,
+            -- and the search settles one lower. With a pre-settled set it does
+            -- not fail, the whole remainder goes out in one step, and the head
+            -- is then left in FanoutProgress with nothing to distribute and no
+            -- way to burn its tokens. That is a pre-existing gap in HeadLogic,
+            -- not something this bound can fix.
             findFittingFanoutTx
               tracer
               wallet
@@ -600,6 +619,16 @@ prepareTxToPost timeHandle ctx spendableUTxO tx =
     upperBoundSlot <- throwLeft $ slotFromUTCTime upperBoundTime
     pure (upperBoundSlot, upperBoundTime)
 
+-- | The single transaction to try before falling back to chunked partial
+-- fanouts, given how many outputs it distributes. 'Nothing' above
+-- 'Accumulator.deployedFanoutBatchSize', where it could only be rejected:
+-- building it means a membership proof over the whole set, discarded on every
+-- step.
+preferredFanoutTx :: Int -> Either err Tx -> Maybe Tx
+preferredFanoutTx numOutputs preferred
+  | numOutputs > Accumulator.deployedFanoutBatchSize = Nothing
+  | otherwise = rightToMaybe preferred
+
 -- | Binary search for the largest chunk size in @[1..maxChunk]@ for which
 -- 'tryTx' returns 'Just'. Assumes the predicate is monotone: if size @n@ fits,
 -- all sizes @< n@ also fit. Uses upper-mid so the search terminates correctly
@@ -656,7 +685,7 @@ fitsTx tracer withinSizeLimits evalCosts evalUTxO tx = do
 -- fits, minimising the number of fanout steps.
 --
 -- Error mapping:
---   * 'StaleChainState' from 'partialFanout' → 'StalePartialFanoutTx' (race
+--   * 'StaleChainState' from 'preparePartialFanout' → 'StalePartialFanoutTx' (race
 --     condition; HeadLogic silently ignores it and the chain observation loop
 --     triggers the correct next step).
 --   * Any other 'PartialFanoutError' → 'FailedToConstructPartialFanoutTx'
@@ -683,9 +712,12 @@ findFittingFanoutTx ::
   --   final/full fanout fallback this is @size - 1@ (the preferred tx handles
   --   the full set; a partial fanout must leave at least one output). For an
   --   explicit non-final partial fanout this is the full @size@: the selection
-  --   is always a strict subset of the head's remaining UTxO, so even
+  --   is normally a strict subset of the head's remaining UTxO, so even
   --   distributing all of it leaves the unselected remainder in the accumulator
-  --   and 'mustNotBeLastBatch' holds.
+  --   and 'mustNotBeLastBatch' holds (see the caller for the one case where it
+  --   does not, which costs a rejected candidate but not the answer).
+  --   The search caps this at 'Accumulator.deployedFanoutBatchSize' regardless:
+  --   no larger subset can be verified, however cheap its transaction is.
   Int ->
   -- | Contestation deadline as SlotNo
   SlotNo ->
@@ -699,12 +731,30 @@ findFittingFanoutTx tracer TinyWallet{evaluateScriptCosts, isTxWithinSizeLimits}
    where
     tryPreferred tx = fits tx >>= bool findFallback (pure (Right tx))
 
-  findFallback = findLargestFitting tryChunk maxChunkSize
+  -- Reading the head output and verifying its accumulator does not depend on the
+  -- chunk size, so it happens once here rather than per candidate. There is
+  -- nothing to prepare when no chunk size is in range: preparing anyway would
+  -- turn a stale chain state into 'StalePartialFanoutTx', which HeadLogic
+  -- ignores, where there is in fact no chunk to post and the caller should hear
+  -- the terminal 'FailedToConstructPartialFanoutTx'. Note that HeadLogic only
+  -- reverts the head on that error while nothing has been distributed yet, so
+  -- this restores the revert on the 'FanoutTx' path and, on the
+  -- 'FinalPartialFanoutTx' path, only the client error.
+  findFallback
+    | searchRange < 1 = pure (Left ())
+    | otherwise = do
+        plan <- orThrow $ preparePartialFanout spendableUTxO seedTxIn proofUTxO fullUTxO
+        findLargestFitting (tryChunk plan) searchRange
    where
-    tryChunk n = buildTx n >>= \tx -> bool Nothing (Just tx) <$> fits tx
+    searchRange = min maxChunkSize Accumulator.deployedFanoutBatchSize
+
+    tryChunk plan n = buildTx plan n >>= \tx -> bool Nothing (Just tx) <$> fits tx
+
+  buildTx plan n =
+    orThrow $ partialFanoutFromPlan ctx plan n deadlineSlot
 
-  buildTx n =
-    either handleErr pure $ partialFanout ctx spendableUTxO seedTxIn n proofUTxO fullUTxO deadlineSlot
+  orThrow :: Either PartialFanoutError a -> m a
+  orThrow = either handleErr pure
    where
     handleErr err = do
       traceWith tracer PartialFanoutFailed{reason = show err}