Output data cleaner

[nhduongvn]

Status

not-tested

Description

Summary

Implements block-by-block cleanup of wma_prod Rucio replication rules for output data during the disk/tape transfer phase, releasing origin-T2 disk space as individual blocks complete rather than waiting for all transfers across the entire workflow to finish.

Background: wma_prod container-level rules are disabled. Block-level rules (one per output block) are used instead, with a configurable number of replicas (blockRuleParams, currently set to copies=1 to pin the origin-T2 replica only — previou sly copies=2). Because there is no container rule, a significant amount of data can be transferred before any individual disk rule reaches OK state — making it necessary to track disk rule completion (TransferDisk) separately from tape rule completion (TransferTape) before allowing full cleanup to proceed.

Changes

MSRuleCleaner.py

• Added cleanOutputBlockRules() — runs during partial transfer phase, expires a wma_prod block rule (lifetime=0) as soon as all files in that block have OK locks in all wmcore_output rules (disk + tape intersection)
• Added _getOkFilesFromRule() helper returning the set of fully-replicated files for a Rucio rule
• Extended _dispatchWflow condition to trigger partial cleanup when either disk or tape transfers are still pending

MSRuleCleanerWflow.py

• Added TransferDisk flag to track when all wmcore_output disk rules are in OK state. The container level of wma_prod is disabled so there might be significant data transfer before the disk rule is OK. Therefore, we need to track this separately — otherwise cleanuplines would proceed before transfers are complete.

WMAgentConfig.py / RucioInjectorPoller.py

• Changed blockRuleParams = {"copies": 1} (configurable) — block rules now pin the origin-T2 replica only, no second site replica

Rucio.py

• Added listReplicaLocks() to retrieve per-file lock states for a given rule ID

MSRuleCleanert.py

• Updated existing test expected dicts for new TransferDisk and WmcOutputRulesMap fields
• Added 7 new unit tests for cleanOutputBlockRules

Is it backward compatible (if not, which system it affects?)

YES