[Store] feat: implements strict multi-tenant quota admission

[Lin-z-w]

Description

Part of #2153. Implements strict multi-tenant quota admission for Mooncake Store.

This PR replaces the inherited/default tenant quota model with a connector-backed policy source:

• Adds enable_multi_tenants, tenant_quota_connector_type, and tenant_quota_connector_uri.
• Adds TenantQuotaPolicyStore and a v1 YAML file connector with atomic writes.
• Requires write/allocate requests to use an explicitly registered tenant when multi-tenant mode is enabled.
• Removes implicit default quota inheritance and the default quota admin endpoint.
• Persists admin tenant quota changes to the connector before applying them in memory.
• Restores runtime metadata from snapshots while loading tenant policy from the connector as source of truth.
• Creates orphan in-memory quota state for metadata tenants missing from policy, allowing reads/removes while blocking new writes.
• Fixes tenant quota policy update/delete races, YAML round-trip safety, and tenant eviction shard scan fairness.

Module

• Transfer Engine (mooncake-transfer-engine)
• Mooncake Store (mooncake-store)
• Mooncake EP (mooncake-ep)
• Mooncake PG (mooncake-pg)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• Common (mooncake-common)
• Mooncake RL (mooncake-rl)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement
• Other

How Has This Been Tested?

Test commands:

cmake --build build --target tenant_quota_test master_service_tenant_quota_test
ctest --test-dir build -R "tenant_quota_test|master_service_tenant_quota_test" --output-on-failure

Test results:

• Unit tests pass
• Integration tests pass (if applicable)
• Manual testing done (describe below)

tenant_quota_test and master_service_tenant_quota_test passed locally.

Checklist

• I have performed a self-review of my own code
• I have formatted my code using ./scripts/code_format.sh
• I have run pre-commit run --all-files and all hooks pass
• I have updated the documentation (if applicable)
• I have added tests to prove my changes are effective
• For changes >500 LOC: I have filed an RFC issue

AI Assistance Disclosure

• No AI tools were used
• AI tools were used (specify below)

Codex was used to help implement, review, test, and document this change. I reviewed and edited the generated code before submission.

[gemini-code-assist]

Code Review

This pull request refactors the tenant quota management system to support a strict multi-tenant mode with connector-managed policies (e.g., YAML files), removing default fallback quotas and requiring explicit tenant registration. Key improvements include the introduction of a TenantQuotaPolicyStore for persistent policy management, updated admin APIs, and modified snapshot restore behavior. The review feedback highlights several optimization opportunities in MasterService, including pre-grouping tenant quotas by shard in ApplyTenantQuotaPolicies to reduce complexity, performing the TenantHasObjects check earlier in DeleteTenantQuotaPolicy to simplify rollback logic, and consolidating loops in RecomputeTenantEffectiveQuotas to avoid redundant double-locking of shard mutexes.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 85.55377% with 180 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
mooncake-store/src/master_service.cpp	79.09%	83 Missing ⚠️
mooncake-store/src/tenant_quota_policy_store.cpp	67.52%	76 Missing ⚠️
...e-store/tests/master_service_tenant_quota_test.cpp	96.79%	11 Missing ⚠️
mooncake-store/include/master_config.h	75.00%	4 Missing ⚠️
mooncake-store/src/master.cpp	60.00%	4 Missing ⚠️
mooncake-store/src/master_admin_service.cpp	66.66%	1 Missing ⚠️
mooncake-store/src/tenant_quota.cpp	93.75%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[yokinoshitayoki]

This leaves a race for zero-charge writes. When replica_num == 0 && nof_replica_num > 0, reserved_quota_charge is 0, so ReserveTenantQuota() only validates registration and does not leave any reserved state behind. A concurrent DeleteTenantQuotaPolicy() can then observe the tenant as empty, delete the policy, and this PutStart can still create metadata afterward at line 2760. That violates the strict registration guarantee and can create a new orphan object after the tenant was deleted.

Can we either keep the tenant policy lock held until metadata_object_count is incremented for zero-charge creates, or add a pending write/metadata marker that makes delete see the tenant as non-empty?

[Lin-z-w]

addressed in the latest revision. The three functions now normalize the tenant before taking snapshot_mutex_, so they no longer acquire tenant_quota_policy_mutex_ while holding snapshot_mutex_.

[yokinoshitayoki]

This introduces an inconsistent lock order. AddReplica() takes tenant_quota_policy_mutex_ before snapshot_mutex_, but this path takes snapshot_mutex_ first and then calls NormalizeTenantIdForWrite(), which takes tenant_quota_policy_mutex_. The same pattern also appears in CreateCopyTask() and CreateMoveTask().

With a pending snapshot writer, this can deadlock on writer-preferring shared_mutex implementations: one thread holds the policy mutex and waits for a snapshot shared lock, while this thread holds a snapshot shared lock and waits for the policy mutex. Can we move tenant normalization before acquiring snapshot_mutex_, as PutStart, CopyStart, and MoveStart already do?

[Lin-z-w]

Agreed. I kept the zero-charge path protected by holding tenant_quota_policy_mutex_ only around the metadata-create admission window, and used metadata_object_count to make DeleteTenantQuotaPolicy observe zero-charge objects. This avoids holding the policy mutex through quota accounting itself while still preventing the tenant-delete race.