Skip to content

tdutils: mmap-backed BufferSlice for large file reads#2303

Open
Lordron wants to merge 1 commit into
ton-blockchain:testnetfrom
Lordron:mmap-state-read-testnet
Open

tdutils: mmap-backed BufferSlice for large file reads#2303
Lordron wants to merge 1 commit into
ton-blockchain:testnetfrom
Lordron:mmap-state-read-testnet

Conversation

@Lordron

@Lordron Lordron commented Apr 16, 2026

Copy link
Copy Markdown

During initial sync, DownloadShardState reads the persistent state file
via td::read_file, which allocates the entire file size as one heap
buffer. For current mainnet masterchain state this is ~7 GB held in RSS
across deserialization.

This change adds BufferAllocator::create_reader_mmap(fd, size) that
mmap's the file MAP_PRIVATE and constructs a BufferRaw whose data
pointer points into the mmap'd region. dec_ref_cnt handles munmap on
the last reference. A new td::read_file_mmap() helper wraps this, and
ReadFile::start_up opts into mmap for whole-file reads of 64 MB or
more (offset=0, size=-1).

Existing heap-backed code paths are untouched; BufferRaw adds a single
nullable pointer that defaults to null, and all callers access through
a new BufferRaw::data() accessor that picks between the inline array
and the external pointer.

Measured on a 32 GB host syncing mainnet masterchain state 60660942:
RSS at +5 min uptime: 30 GB -> 9 GB
Swap used at peak: 15 GB -> 65 MB
CPU usage: unchanged
Disk reads: +1 MB/s from mmap page faults (9% util)
DB persistence rate: unchanged

The 64 MB threshold avoids mmap overhead for small files while capturing
the target case of multi-GB persistent state files.

During initial sync, DownloadShardState reads the persistent state file
  via td::read_file, which allocates the entire file size as one heap
  buffer. For current mainnet masterchain state this is ~7 GB held in RSS
  across deserialization.

  This change adds BufferAllocator::create_reader_mmap(fd, size) that
  mmap's the file MAP_PRIVATE and constructs a BufferRaw whose data
  pointer points into the mmap'd region. dec_ref_cnt handles munmap on
  the last reference. A new td::read_file_mmap() helper wraps this, and
  ReadFile::start_up opts into mmap for whole-file reads of 64 MB or
  more (offset=0, size=-1).

  Existing heap-backed code paths are untouched; BufferRaw adds a single
  nullable pointer that defaults to null, and all callers access through
  a new BufferRaw::data() accessor that picks between the inline array
  and the external pointer.

  Measured on a 32 GB host syncing mainnet masterchain state 60660942:
    RSS at +5 min uptime: 30 GB -> 9 GB
    Swap used at peak:   15 GB -> 65 MB
    CPU usage:           unchanged
    Disk reads:          +1 MB/s from mmap page faults (9% util)
    DB persistence rate: unchanged

  The 64 MB threshold avoids mmap overhead for small files while capturing
  the target case of multi-GB persistent state files.
@DanShaders DanShaders self-assigned this Apr 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants