Database Optimizations#3294
Open
ouziel-slama wants to merge 33 commits into
Open
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## develop #3294 +/- ##
===========================================
- Coverage 91.34% 89.11% -2.24%
===========================================
Files 113 117 +4
Lines 15608 16100 +492
===========================================
+ Hits 14257 14347 +90
- Misses 1351 1753 +402 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…DF calls, and streamline rowtracer
…gration in --api-only mode
Codecov Report❌ Patch coverage is ❌ Your patch check has failed because the patch coverage (57.30%) is below the target coverage (90.00%). You can increase the patch coverage or adjust the target coverage.
Additional details and impacted files@@ Coverage Diff @@
## develop #3294 +/- ##
============================================
- Coverage 91.38% 54.62% -36.77%
============================================
Files 114 116 +2
Lines 16296 17132 +836
============================================
- Hits 14892 9358 -5534
- Misses 1404 7774 +6370 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This was referenced May 15, 2026
…dex) pairs Drop the 129-char tx0hash_tx1hash TEXT id on order_matches, bet_matches, rps_matches and the *_match_id FK on the *_match_expirations, btcpays, rpsresolves and bet_match_resolutions tables; key matches by the integer (tx0_index, tx1_index) pair instead. The textual id is reconstructed on read at the API and consensus boundaries, so the public API and consensus hashes are unchanged.
Add a surrogate `asset_index INTEGER PRIMARY KEY` to `assets` (assigned in issuance order; the protocol `asset_id` stays TEXT) and replace every asset-name TEXT column with that compact integer FK across balances, credits, debits, sends, orders, order_matches, issuances, dividends, dispensers, dispenses, dispenser_refills, fairminters, fairmints, destructions, pools and pool_*. Folded into migration 0010. Asset names stay visible everywhere with no consensus change: - the rowtracer decodes asset_index -> name on read; - events._prepare_record_for_insert encodes name -> index on write (returning a copy, so OrdersCache still gets names); - the consensus journal keeps the original names, so ledger_hash and messages_hash are unchanged; - ledger query functions and the API select_rows resolve name -> index for WHERE filters (select_rows only on Ledger DB, via a main.assets probe; the State DB consolidated tables keep names); - the API migrations (0004/0006/0014) decode index -> name while ledger_db is attached. lp_asset stays TEXT (the LP token is referenced before it exists). Invalid records referencing a never-registered asset store NULL. Order-sensitive consensus paths resolve the asset *name* for ordering so the pre-normalization processing order is preserved: get_dispensers (order_by=asset) and get_address_balances, the latter feeding the msg_index / debit / credit sequence in move, detach and sweep.
…olumns Finishes the storage-compaction work (issue #2744 points 3 & 4), folded into the unreleased migration 0010.compact_hash_storage, mirroring the asset_index normalization. Address normalization: - New address_list(address_id INTEGER PRIMARY KEY, address TEXT UNIQUE) identity table; every address column (source, destination, address, issuer, feed_address, tx0/tx1_address, winner, oracle_address, origin, *_address, utxo_address) becomes an INTEGER address_id FK. - Transparent: the rowtracer decodes address_id -> string on read and the write path encodes string -> id (ensure_address, first-seen-on-use), so consensus hashes and the public API are unchanged. The State DB keeps address strings (consolidation decodes while ledger_db is attached). The existing addresses options table is preserved and FK'd; mempool stays TEXT. - Per-connection address caches are bounded (LRU) since addresses number in the millions, and are reset on mempool/reorg rollback. UTXO compaction: - balances/credits/debits utxo TEXT (tx_hash:vout) -> (utxo_tx_hash BLOB, utxo_vout INTEGER). The tx_hash is stored as a raw BLOB rather than a tx_index FK because an attach destination may be any valid bitcoin UTXO whose tx is absent from the transactions table; the utxo string is reconstructed transparently by the rowtracer. Also: API prepare_where address/utxo filter rewrites, State DB consolidation decode (0004/0006/0014), consensus-ordering fixes (get_asset_balances subquery shadowing, holders() utxo order, held() dedup separator), and test/mock updates. Full unit suite green (1507 passed).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.