disagg: limit S3 read amplification on compute nodes (#10771)#10790

Merged

ti-chi-bot[bot] merged 37 commits into

pingcap:release-nextgen-202603from

ti-chi-bot:cherry-pick-10771-to-release-nextgen-202603

Apr 8, 2026

Commits on Apr 8, 2026

disagg: add configurable node-level S3 read limiter

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: throttle S3 readers and deduplicate cache misses

JaySon-Huang
authored and
ti-chi-bot
committed
Refine metrics and comments

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: add remote cache phase-2 metrics and coverage

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: remove duplicate S3 active stream metric

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: share file cache wait implementation

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: clarify file cache get paths

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: handle failed bounded wait downloads

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: reuse copyData in limited cache downloads

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: clarify conservative S3 read charging

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: share S3 random access read and seek tails

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: extend S3 read limiter unit coverage

JaySon-Huang
authored and
ti-chi-bot
committed
Refine metrics

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: enrich file cache observability

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: remove S3 stream cap support

JaySon-Huang
authored and
ti-chi-bot
committed
docs: add English disagg S3 backpressure design

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: clarify shared limiter test semantics

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: keep S3 limiter object across disable reload

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: address review follow-up fixes

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: avoid large-request stalls in S3 limiter

JaySon-Huang
authored and
ti-chi-bot
committed
Format codes

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: decouple S3 read accounting from throttling

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: add mark and minmax cache eviction APIs

JaySon-Huang
authored and
ti-chi-bot
committed
Remove unnecessary design doc

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: address cache eviction and filecache review notes

JaySon-Huang
authored and
ti-chi-bot
committed
Update grafana panel

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: tighten filecache startup and low-limit reads

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: fix failed download cache accounting

JaySon-Huang
authored and
ti-chi-bot
committed
Update grafana panel

JaySon-Huang
authored and
ti-chi-bot
committed
Update grafana panel

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: refine filecache failure cleanup metrics

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: restore S3 fast paths and scope bg download failure

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: restore disabled-limiter seek fast path

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: clean up failed file cache scheduling

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: clarify S3 limiter semantics and harden tests

JaySon-Huang
authored and
ti-chi-bot
committed
disagg: avoid pool access during file cache shutdown

JaySon-Huang
authored and
ti-chi-bot
committed
tests: support overriding HUB_ADDR/PD_IMAGE/TIKV_IMAGE/TIDB_IMAGE via env vars in next-gen fullstack test
JaySon-Huang
committed