Skip to content

Extend FS buffer prefetch to async multi-buffer path#14552

Open
eniac1024 wants to merge 6 commits intofacebook:mainfrom
eniac1024:jycai/support-fs-buffer-for-async-prefetch
Open

Extend FS buffer prefetch to async multi-buffer path#14552
eniac1024 wants to merge 6 commits intofacebook:mainfrom
eniac1024:jycai/support-fs-buffer-for-async-prefetch

Conversation

@eniac1024
Copy link
Copy Markdown
Contributor

@eniac1024 eniac1024 commented Apr 2, 2026

Problem

RocksDB's FilePrefetchBuffer supports an optimization where the filesystem provides its own buffer (fs_scratch), avoiding an extra memory copy. However, this optimization was restricted to the synchronous single-buffer path (num_buffers_ == 1). The async multi-buffer prefetch path — used for scan-ahead with 2+ buffers — always allocated its own buffers and copied data, missing the zero-copy benefit.

Solution

Remove the num_buffers_ == 1 gate from UseFSBuffer() and plumb use_fs_buffer through the entire async pipeline:

  • ReadAsync: sets scratch = nullptr to signal the FS to provide its own buffer; sync fallback routes to FSBufferDirectRead
  • PrefetchAsyncCallback: adopts FS-provided buffer via SetBuffer() instead of updating pre-allocated buffer size
  • HandleOverlappingAsyncData: handles the single-buffer-remaining case for FS buffers by polling async reads and copying partial data to overlap_buf_; passes use_fs_buffer through the multi-buffer path
  • PrefetchAsync / PrefetchRemBuffers: compute use_fs_buffer once and thread it through all ReadAheadSizeTuning, PrepareBufferForRead, and ReadAsync calls

Tests are updated to exercise FS buffer mode in async paths, with two new tests (PrefetchAsyncWithFSBuffer, SequentialReadWithFSBuffer) validating correctness.

file/file_prefetch_buffer.h

  • Removed num_buffers_ == 1 restriction from UseFSBuffer()

  • Added use_fs_buffer parameter to ReadAsync() and PrefetchRemBuffers()

  • Updated comment on UseFSBuffer() to remove stale "async not handled" note

file/file_prefetch_buffer.cc

  • ReadAsync: sets scratch = nullptr when use_fs_buffer; sync fallback routes to FSBufferDirectRead

  • PrefetchAsyncCallback: added fs_scratch branch adopting FS buffer via SetBuffer() with offset_/initial_end_offset_ update

  • HandleOverlappingAsyncData: added single-buffer FS path — polls async read if in progress, copies partial data to overlap_buf_; multi-buffer path computes and passes use_fs_buffer instead of hardcoded false

  • PrefetchInternal: added num_buffers_ == 1 guard on HandleOverlappingSyncData; replaced redundant UseFSBuffer(reader) call with cached local

  • PrefetchAsync: computes use_fs_buffer once and threads it through all ReadAheadSizeTuning, PrepareBufferForRead, and ReadAsync calls

  • PrefetchRemBuffers: accepts and threads use_fs_buffer to ReadAheadSizeTuning and ReadAsync

file/prefetch_test.cc

  • SetUp always uses BufferReuseFS (previously skipped for async)

  • Added ReadAsync override in mock FS with fs_scratch allocation (scratch == nullptr check) and io_handle/del_fn set to nullptr

  • Updated FSBufferPrefetchStatsInternals with separate use_fs_buffer assertion branches for async path

  • Refactored FSBufferPrefetchUnalignedReads — unified sync/async assertions where FS buffer alignment = 1 makes them identical

  • New test: PrefetchAsyncWithFSBuffer — async prefetch at non-zero offset, num_buffers = 2

  • New test: SequentialReadWithFSBuffer — two consecutive sequential reads with both sync and async paths

db/db_io_failure_test.cc

  • Added ReadAsync override in CorruptionFS with fs_scratch allocation for FS buffer mode

  • IterReadCorruptionRetry: skips fs_buffer + async_io combination (read ordering changes cause corruption to hit wrong read)

@meta-cla meta-cla Bot added the CLA Signed label Apr 2, 2026
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 2, 2026

✅ clang-tidy: No findings on changed lines

Completed in 179.7s.

@eniac1024 eniac1024 force-pushed the jycai/support-fs-buffer-for-async-prefetch branch 3 times, most recently from 52651ee to c01cbe7 Compare April 14, 2026 05:19
@eniac1024 eniac1024 force-pushed the jycai/support-fs-buffer-for-async-prefetch branch from c01cbe7 to 14ac8df Compare April 14, 2026 06:18
@eniac1024 eniac1024 force-pushed the jycai/support-fs-buffer-for-async-prefetch branch from 02b544d to 3946458 Compare April 14, 2026 07:11
@eniac1024 eniac1024 marked this pull request as draft April 27, 2026 09:54
@eniac1024 eniac1024 marked this pull request as ready for review April 27, 2026 09:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant