feat: Add migration backfill for mptoken_issuance_history by BryanJ1ang · Pull Request #3128 · XRPLF/clio

BryanJ1ang · 2026-06-30T16:56:37Z

Note: this PR depends on previous ETL indexing PR here

Summary

Adds MPTTransactionHistoryMigrator, a Cassandra migration that backfills historical MPT issuance transaction history into the new MPT index tables:

mptoken_issuance_transactions
account_mptoken_issuance_transactions

The migrator full-scans the transactions table, reuses the live ETL MPT extractor, writes the same deterministic index rows produced by live indexing, and waits for queued async writes before marking the migration as complete.

Details

Registers MPTTransactionHistoryMigrator with the Cassandra migration manager.
Uses migration scan settings for full-table scanning:
- migration.full_scan_threads
- migration.full_scan_jobs
- migration.cursors_per_job
Adds Cassandra driver paging support so token-range scans read all result pages.
Makes migration scans select explicit columns instead of relying on table column order.
Propagates full-table scanner worker failures so failed scans do not get marked migrated.
Keeps the migration non-blocking for Clio startup.

Testing

Added coverage for:

Migrator status before and after running.
Backfilling MPT issuance create paths and multi-issuance transaction fan-out.
Idempotent reruns.
Empty transaction tables.
Parity between migration backfill rows and live ETL rows.
Failed transactions with explicit MPT issuance references.
Cassandra paging across more than one driver page.
Full-table scanner validation and worker error propagation.

codecov · 2026-07-07T18:05:08Z

Codecov Report

❌ Patch coverage is 38.09524% with 65 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
.../migration/cassandra/CassandraMigrationBackend.hpp	0.00%	34 Missing ⚠️
...ration/cassandra/MPTTransactionHistoryMigrator.cpp	0.00%	25 Missing ⚠️
...c/migration/cassandra/impl/TransactionsAdapter.cpp	72.72%	1 Missing and 2 partials ⚠️
src/migration/MigrationApplication.cpp	60.00%	1 Missing and 1 partial ⚠️
src/main/Main.cpp	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

BryanJ1ang · 2026-07-08T16:22:25Z

/ai-reviewer

Kassaking7

You may also want to add missing unit tests based on the codecov report

BryanJ1ang · 2026-07-22T05:13:28Z

You may also want to add missing unit tests based on the codecov report

Added some unit tests but for CassandraMigrationBackend.hpp MPTTransactionHistoryMigrator.cpp and MigrationApplication.cpp these can't be unit tested since they need a Cassandra db. They are covered by integration tests.

BryanJ1ang · 2026-07-22T05:14:35Z

@PeterChen13579 @yinyiqian1 I can't add you guys as reviewers but please take a look when you guys can!

Kassaking7

LGTM

yinyiqian1 · 2026-07-27T15:45:48Z

    static constexpr auto kDeleter = [](CassCollection* ptr) { cass_collection_free(ptr); };

-    static void
-    throwErrorIfNeeded(CassError const rc, std::string_view const label)


currently collection and statement is using the refactored function. But there are other places still using the local lambda function. So this refactor is not complete.
I suggest we revert these changes and leave the refactor in a separate PR.

Copilot

Pull request overview

Adds a Cassandra migration to backfill historical MPToken issuance transaction history into the new MPT index tables, reusing the existing live ETL extractor and improving migration full-table scan robustness (paging + error propagation).

Changes:

Introduces MPTTransactionHistoryMigrator and registers it with the Cassandra migration manager.
Enhances full-table scanning: explicit column selection via kSelectColumns, Cassandra driver paging support, and propagation of worker/read failures.
Adds/extends unit + integration coverage around scanner behavior, adapter decoding failures, migrator status semantics, and parity with live ETL indexing.

Reviewed changes

Copilot reviewed 36 out of 36 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/unit/util/BatchingTests.cpp	Adds unit coverage for `util::BatchBuffer` batching behavior and concurrency.
tests/unit/migration/MigratorRegisterTests.cpp	Verifies failed migrators don’t write “Migrated” status.
tests/unit/migration/MigrationApplicationTests.cpp	Adds unit tests for migrator CLI/application behavior via a mock manager.
tests/unit/migration/cassandra/TransactionsAdapterTests.cpp	Tests transaction row deserialization and decode-failure signaling.
tests/unit/migration/cassandra/SpecTests.cpp	Updates table spec test to include explicit select-column lists.
tests/unit/migration/cassandra/FullTableScannerTests.cpp	Updates tests for new scanner wait/error behavior and adds new validations.
tests/unit/CMakeLists.txt	Registers new unit test sources.
tests/integration/migration/cassandra/MPTTransactionHistoryMigratorTests.cpp	End-to-end integration coverage for MPT backfill, idempotency, paging, and ETL parity.
tests/integration/migration/cassandra/ExampleTransactionsMigrator.cpp	Updates example migrator to new adapter + scanner wait API.
tests/integration/migration/cassandra/ExampleObjectsMigrator.cpp	Updates example migrator to new scanner wait API.
tests/integration/migration/cassandra/CassandraMigrationManagerTests.cpp	Adds integration test to ensure migration scans read all driver pages.
tests/integration/data/cassandra/BaseTests.cpp	Adds base Cassandra test for paging-state iteration across pages.
tests/integration/CMakeLists.txt	Registers new integration test source.
tests/common/util/MockMigrationManager.hpp	Adds a mock `MigrationManagerInterface` for application-layer tests.
tests/common/migration/TestMigrators.hpp	Adds a throwing migrator used to validate failure semantics.
src/util/Batching.hpp	Adds `util::BatchBuffer` for streaming batching + flush semantics.
src/migration/MigrationApplication.hpp	Adds constructor overload allowing injection of a prebuilt migration manager (tests).
src/migration/MigrationApplication.cpp	Avoids double Prometheus init and implements injected-manager constructor.
src/migration/CMakeLists.txt	Links migration library against `clio_etl` to reuse extractor for backfill.
src/migration/cassandra/MPTTransactionHistoryMigrator.hpp	Declares new backfill migrator for historical MPT issuance tx indexing.
src/migration/cassandra/MPTTransactionHistoryMigrator.cpp	Implements full-table scan + batching + decode-failure handling + durable completion.
src/migration/cassandra/impl/TransactionsAdapter.hpp	Adds explicit select columns + decode-failure callback + logging support.
src/migration/cassandra/impl/TransactionsAdapter.cpp	Implements deserialization with controlled handling of decode failures.
src/migration/cassandra/impl/Spec.hpp	Requires `kSelectColumns` and enforces tuple/column-count alignment.
src/migration/cassandra/impl/ObjectsAdapter.hpp	Adds explicit `kSelectColumns` matching row tuple order.
src/migration/cassandra/impl/FullTableScanner.hpp	Adds settings validation and replaces silent waits with error-propagating join.
src/migration/cassandra/impl/CassandraMigrationSchema.hpp	Prepares full-scan statements using explicit column lists.
src/migration/cassandra/CassandraMigrationManager.hpp	Registers `MPTTransactionHistoryMigrator` for Cassandra migrations.
src/migration/cassandra/CassandraMigrationBackend.hpp	Adds prepared-statement caching, paging-size/state handling, and fail-closed scan errors.
src/main/Main.cpp	Initializes Prometheus in the `migrate` CLI path before creating the migrator app.
src/etl/impl/ext/MPT.cpp	Simplifies issuance-index row counting (removes per-tx warning threshold logic).
src/data/cassandra/impl/Statement.hpp	Adds paging-size/state helpers and centralizes error throwing.
src/data/cassandra/impl/Result.hpp	Adds `hasMorePages()` API.
src/data/cassandra/impl/Result.cpp	Implements `hasMorePages()`.
src/data/cassandra/impl/Collection.hpp	Switches to shared Cassandra error helper (removes local throw helper).
src/data/cassandra/Error.hpp	Adds shared `throwErrorIfNeeded` helper used by statement/binding/paging APIs.

Comments suppressed due to low confidence (1)

src/util/Batching.hpp:111

Same capacity issue as in add(): std::exchange(buffer_, {}) drops buffer_ capacity, so the next accumulation may reallocate. Reserving batchSize_ after clearing helps keep allocation behavior stable across flush cycles.

            std::scoped_lock const lock{mutex_};
            batch = std::exchange(buffer_, {});
        }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+            if (buffer_.size() < batchSize_)
+                return;
+            batch = std::exchange(buffer_, {});
+        }


PeterChen13579

Just realized you have to merge with latest develop which updates to the newest libxrpl. Make sure to do a clean build. I will continue to review after this as merging with latest develop will change a lot

PeterChen13579 · 2026-07-26T04:29:48Z

-                );
-            }
+        auto throwBindingErrorIfNeeded = [idx](CassError rc, std::string_view label) {
+            if (rc != CASS_OK)


You're checking if (rc != CASS_OK) twice, because throwErrorIfNeeded also does if (rc != CASS_OK) check

yinyiqian1 · 2026-07-27T16:33:26Z

+    setPagingSize(std::int32_t const pageSize) const
+    {
+        auto const rc = cass_statement_set_paging_size(*this, pageSize);
+        throwErrorIfNeeded(rc, "Set paging size");


the refactor of throwErrorIfNeeded is mainly for page.
We can keep the logic within Statement.hpp, instead of refactoring in error.hpp, collection.hpp, etc.

yinyiqian1 · 2026-07-27T16:50:41Z

-                    fmt::format("[{}] at idx {}: {}", label, idx, cass_error_desc(rc))
-                );
-            }
+        auto throwBindingErrorIfNeeded = [idx](CassError rc, std::string_view label) {


The change in bindAt does not change anything. The error message is almost the same after this change, just the ]'s location is moved.

before change: [...] at idx ..
after change: [... at idx ..]

Statement.hpp and Collection.hpp each carried a private static copy of the same CassError-to-logic_error helper; both now use one shared free function in data::cassandra::impl. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…nst ref A transaction that fails to deserialize is a deterministic data error, so rerunning the migration could never get past it; log and skip the row instead of permanently wedging the backfill. Callback and DB read errors still fail closed. The scan callback now takes STTx/TxMeta by const ref, removing a deep copy per scanned row. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Buffer extracted records across transactions and flush in batches (mirroring the live ETL path's per-ledger accumulation) so the backend coalesces them into full-size write batches instead of submitting one tiny write pair per transaction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The warning was not operator-actionable and per-tx fanout beyond the threshold should not occur in practice; the Prometheus counter still tracks index rows written. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Key the prepared-statement cache by table name alone (partition key and select columns are fixed per TableDesc), replace the rowsRead counter with a bool, and author the read-failure message once for both the log and the thrown exception. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The TableSpec concept now requires the kSelectColumns list to have exactly one column per Row tuple element, so the two cannot silently drift when columns are added or removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

yinyiqian1 · 2026-07-27T21:14:49Z

-        }
-
-        indexRowsWritten += txIndexRows;
+            indexRowsWritten += 1 + record.accounts.size();


I don't think the change in MPT.cpp is needed

yinyiqian1 · 2026-07-27T21:49:52Z

+    /**
+     * @brief The full table scanner settings.
+     */
+    struct FullTableScannerSettings {


this struct does not need to be moved

yinyiqian1 · 2026-07-27T21:50:48Z

+
+private:
+    [[nodiscard]] static std::uint32_t
+    validatedCtxThreadsNum(FullTableScannerSettings const& settings)


this is not needed

yinyiqian1 · 2026-07-27T21:50:56Z

+    }
+
+    [[nodiscard]] static std::size_t
+    computeCursorsNum(FullTableScannerSettings const& settings)


This is not needed

yinyiqian1 · 2026-07-27T21:53:21Z

    FullTableScanner(FullTableScannerSettings settings, TableAdapter&& reader)
-        : ctx_(ExecutionContextType(settings.ctxThreadsNum))
-        , cursorsNum_(settings.jobsNum * settings.cursorsPerJob)
+        : ctx_(ExecutionContextType(validatedCtxThreadsNum(settings)))


we don't need to refactor to pass in validatedCtxThreadsNum

yinyiqian1 · 2026-07-27T21:53:55Z

-        );
-
-        auto const cursors = TokenRangesProvider{cursorsNum_}.getRanges();
+        auto const cursors =


We don't need the refactor.

Restores Error.hpp, Collection.hpp and Statement.hpp::bindAt to their develop state. The shared throwErrorIfNeeded helper extracted into Error.hpp was an unrelated dedup, and it left four other call sites (Result.hpp, Tuple.hpp x2, Cluster.cpp) on the old inline pattern, so it did not deliver the consistency it aimed for. That cleanup belongs in its own PR that converts every site and settles on one message format. Reverting bindAt also drops two incidental changes this branch had made to it: an eagerly-evaluated fmt::format on every successful bind, and a change to the error message shape from "[label] at idx N" to "[label at idx N]". Drop the private static throwErrorIfNeeded too, and inline its check into the two paging methods that were its only callers. Keeping it would have added a sixth copy of the same check-and-throw to this subsystem, which is the duplication the reverted commit set out to remove. Removing it also leaves bindAt's local lambda with no class-scope name to shadow. Statement.hpp now differs from develop only by setPagingSize and setPagingState, both of which the MPT backfill token-range scan requires. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Restores the per-transaction MPT issuance index fanout warning in MPTExt::writeMPTDataFromTransactions to match develop; the removal was unrelated to the migration work on this branch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 33 out of 33 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

src/migration/MigrationApplication.cpp:48

The injecting constructor accepts a potentially null migrationManager; run()/printStatus()/migrate() dereference migrationManager_ unconditionally, so constructing with nullptr would crash. Add a defensive check (throw/ASSERT) here to fail fast with a clear error.

MigratorApplication::MigratorApplication(
    MigrateSubCmd command,
    std::shared_ptr<migration::MigrationManagerInterface> migrationManager
)
    : migrationManager_(std::move(migrationManager)), cmd_(std::move(command))
{
}

BryanJ1ang force-pushed the brjiang-mpt-rpc-migration branch 2 times, most recently from a0b2942 to 891846e Compare July 7, 2026 16:59

BryanJ1ang marked this pull request as ready for review July 8, 2026 16:16

Kassaking7 reviewed Jul 20, 2026

View reviewed changes

Comment thread src/data/cassandra/impl/Statement.hpp Outdated

Kassaking7 reviewed Jul 20, 2026

View reviewed changes

Comment thread src/migration/cassandra/impl/TransactionsAdapter.cpp Outdated

BryanJ1ang force-pushed the brjiang-mpt-rpc-migration branch from 2565b28 to 9e0d9fe Compare July 22, 2026 02:26

yinyiqian1 self-requested a review July 22, 2026 20:55

PeterChen13579 self-requested a review July 23, 2026 16:20

BryanJ1ang requested a review from Kassaking7 July 24, 2026 03:21

Kassaking7 approved these changes Jul 24, 2026

View reviewed changes

yinyiqian1 reviewed Jul 27, 2026

View reviewed changes

mathbunnyru requested a review from Copilot July 27, 2026 15:52

Copilot started reviewing on behalf of mathbunnyru July 27, 2026 15:52 View session

Copilot AI reviewed Jul 27, 2026

View reviewed changes

Comment thread src/util/Batching.hpp

Comment on lines +92 to +95

if (buffer_.size() < batchSize_)

return;

batch = std::exchange(buffer_, {});

}

PeterChen13579 requested changes Jul 27, 2026

View reviewed changes

yinyiqian1 reviewed Jul 27, 2026

View reviewed changes

BryanJ1ang and others added 9 commits July 27, 2026 11:15

feat: Migration backfill for mp token issuance history

bfa2fa7

Read all driver pages in MPT backfill token-range scan

04ad850

refactor: use explicit columns for Cassandra migration scans

9963737

Clang format

ea956b2

Remove MPT index fanout warning

f1feb7e

The warning was not operator-actionable and per-tx fanout beyond the threshold should not occur in practice; the Prometheus counter still tracks index rows written. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

BryanJ1ang and others added 7 commits July 27, 2026 11:15

Add unit tests for MigrationApplication

81f0bfd

extract reusable BatchBuffer from MPT migrator

f63d7ff

Only format bind error message on failure

e64db6e

Added unit tests

39ed47d

fail backfill closed on undecodable transactions

c45a052

Streamlined comments

0e2592d

yinyiqian1 reviewed Jul 27, 2026

View reviewed changes

BryanJ1ang force-pushed the brjiang-mpt-rpc-migration branch from 29a8354 to 03dd0ca Compare July 28, 2026 02:11

BryanJ1ang and others added 3 commits July 27, 2026 19:12

Revert FullTableScanner

6914780

BryanJ1ang force-pushed the brjiang-mpt-rpc-migration branch from 03dd0ca to 6914780 Compare July 28, 2026 02:12

BryanJ1ang requested a review from PeterChen13579 July 28, 2026 08:04

mathbunnyru requested a review from Copilot July 28, 2026 15:11

Copilot started reviewing on behalf of mathbunnyru July 28, 2026 15:12 View session

Copilot AI reviewed Jul 28, 2026

View reviewed changes

Uh oh!

Conversation

BryanJ1ang commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Testing

Uh oh!

codecov Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

BryanJ1ang commented Jul 8, 2026

Uh oh!

Kassaking7 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

BryanJ1ang commented Jul 22, 2026

Uh oh!

BryanJ1ang commented Jul 22, 2026

Uh oh!

Kassaking7 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

PeterChen13579 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

BryanJ1ang commented Jun 30, 2026 •

edited

Loading

codecov Bot commented Jul 7, 2026 •

edited

Loading

PeterChen13579 left a comment •

edited

Loading