Skip to content

🤺 fix: Prefix MCP Tool Cache Write Fences #17458

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
danny-avila merged 5 commits into from
Oct 26, 2026
+697 −31
Merged

🤺 fix: Prefix MCP Tool Cache Write Fences #17458

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c4f04b1
fix: Prefix MCP tool-cache ownership fences
lia-librechat Oct 26, 2026
c1ff264
test: Keep fence-prefix regressions lint clean
lia-librechat Oct 26, 2026
1efd6e1
fix: Preserve fence slots for literal Redis prefix braces
lia-librechat Oct 26, 2026
1c9c70b
fix: Bridge MCP ownership fences across rolling upgrades
lia-librechat Oct 26, 2026
b346a83
fix: Isolate deployment ownership in the legacy fence bridge
lia-librechat Oct 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 56 additions & 3 deletions api/server/services/Config/__tests__/getCachedTools.lock.spec.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ const getLogStores = require('~/cache/getLogStores');
const mockCache = { get: jest.fn(), set: jest.fn(), delete: jest.fn() };
getLogStores.mockReturnValue(mockCache);

const { createMCPCatalogStore } = require('@librechat/api');
const {
getCachedTools,
updateCachedGlobalTools,
Expand Down Expand Up @@ -62,6 +63,58 @@ describe('global tool cache write lock', () => {
mockCache.delete.mockResolvedValue(true);
});

describe.each([
{ REDIS_KEY_PREFIX: undefined },
{ REDIS_KEY_PREFIX: '' },
{ REDIS_KEY_PREFIX: 'myprefix_' },
{ REDIS_KEY_PREFIX: 'myprefix_', GLOBAL_PREFIX_SEPARATOR: ':' },
{ REDIS_KEY_PREFIX: 'myprefix_{tenant}', GLOBAL_PREFIX_SEPARATOR: '' },
{ REDIS_KEY_PREFIX: 'myprefix_{' },
{ REDIS_KEY_PREFIX: 'myprefix_}' },
{ REDIS_KEY_PREFIX: 'myprefix_{{tenant}' },
{ REDIS_KEY_PREFIX: 'myprefix_', GLOBAL_PREFIX_SEPARATOR: '{' },
])('fence key namespace %j', (cacheConfig) => {
it.each(['global', 'user'])(
'namespaces the %s fence without changing its slot',
async (scope) => {
mockCache.get.mockResolvedValue(null);
const store = createMCPCatalogStore({
cacheConfig,
getCache: () => mockCache,
ioredisClient: mockRedisClient,
keyvRedisClient: mockKeyvRedisClient,
});
const prefix = cacheConfig.REDIS_KEY_PREFIX
? `${cacheConfig.REDIS_KEY_PREFIX}${cacheConfig.GLOBAL_PREFIX_SEPARATOR ?? '::'}`
: '';
const namespace = `${prefix}${CacheKeys.TOOL_CACHE}:`;
const guardedKey =
scope === 'global'
? `${namespace}tools:global`
: `${namespace}tools:metadata:mcp:user-generation:{user-1:server-1}`;

if (scope === 'global') {
await expect(store.setCachedTools({ builtin: {} })).resolves.toBe(true);
} else {
await expect(
store.getMCPToolsCacheGeneration({ userId: 'user-1', serverName: 'server-1' }),
).resolves.toEqual(expect.any(String));
}

const fenceKey = mockKeyvRedisClient.eval.mock.calls[0][1].keys[0];
expect(fenceKey.startsWith(namespace)).toBe(true);
expect(fenceKey).toContain('write-fence');
expect(fenceKey).not.toBe(guardedKey);
expect(mockKeyvRedisClient.eval.mock.calls.map(([, options]) => options.keys)).toEqual([
[fenceKey],
[fenceKey, guardedKey],
[fenceKey],
]);
expect(calculateSlot(fenceKey)).toBe(calculateSlot(guardedKey));
},
);
});

it('waits for the shared Redis client before accessing the catalog', async () => {
mockCache.get.mockResolvedValue(null);
let resolveRedisReady;
Expand Down Expand Up @@ -147,7 +200,7 @@ describe('global tool cache write lock', () => {
expect.stringContaining("redis.call('GET', KEYS[1])"),
expect.objectContaining({
keys: [
`tools:global:write-fence:{${CacheKeys.TOOL_CACHE}:tools:global}`,
`${CacheKeys.TOOL_CACHE}:tools:global:write-fence:{${CacheKeys.TOOL_CACHE}:tools:global}`,
`${CacheKeys.TOOL_CACHE}:tools:global`,
],
}),
Expand All @@ -168,7 +221,7 @@ describe('global tool cache write lock', () => {
expect.stringContaining("redis.call('PSETEX', KEYS[2]"),
expect.objectContaining({
keys: [
`tools:global:write-fence:{${CacheKeys.TOOL_CACHE}:tools:global}`,
`${CacheKeys.TOOL_CACHE}:tools:global:write-fence:{${CacheKeys.TOOL_CACHE}:tools:global}`,
`${CacheKeys.TOOL_CACHE}:tools:global`,
],
arguments: [
Expand Down Expand Up @@ -339,7 +392,7 @@ describe('global tool cache write lock', () => {
expect.stringContaining("redis.call('EXISTS', KEYS[2])"),
expect.objectContaining({
keys: [
`tools:mcp:write-fence:{user-1:server-1}`,
`${CacheKeys.TOOL_CACHE}:tools:mcp:write-fence:{user-1:server-1}`,
`${CacheKeys.TOOL_CACHE}:tools:metadata:mcp:user-legacy-fence:{user-1:server-1}`,
`${CacheKeys.TOOL_CACHE}:tools:metadata:mcp:user-generation:{user-1:server-1}`,
`${CacheKeys.TOOL_CACHE}:tools:mcp:user:{user-1:server-1}:v2:config-current`,
Expand Down
20 changes: 18 additions & 2 deletions packages/api/src/mcp/catalog/store.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { randomUUID } from 'crypto';
import calculateSlot from 'cluster-key-slot';
import { logger } from '@librechat/data-schemas';
import { CacheKeys, Time } from 'librechat-data-provider';
import type { LCAvailableTools } from '../types';
Expand Down Expand Up @@ -348,9 +349,24 @@ export function createMCPCatalogStore(deps: CatalogStoreDeps): MCPCatalogStore {
const closingBrace = openingBrace >= 0 ? key.indexOf('}', openingBrace + 1) : -1;
return closingBrace > openingBrace + 1 ? key.slice(openingBrace + 1, closingBrace) : key;
};
const prefixHasBraces = /[{}]/.test(rawKey(''));
const ownershipFenceKey = (name: string, guardedKey: string): string => {
let key = rawKey(`${name}:{${redisHashTag(guardedKey)}}`);
if (!prefixHasBraces) return key;
const slot = calculateSlot(guardedKey);
if (calculateSlot(key) === slot) return key;
// Literal prefix braces can override the synthetic tag. Reuse the guarded tag
// when present; otherwise find a same-slot suffix without adding braces.
let suffix = 0;
key = `${guardedKey}:${name}`;
while (calculateSlot(key) !== slot) {
key = `${guardedKey}:${name}:${suffix++}`;
}
return key;
};
const globalCacheRawKey = rawKey(ToolCacheKeys.GLOBAL);
/** Hashes to the same Redis Cluster slot as the unchanged legacy global catalog key. */
const globalFenceKey = `tools:global:write-fence:{${redisHashTag(globalCacheRawKey)}}`;
const globalFenceKey = ownershipFenceKey('tools:global:write-fence', globalCacheRawKey);

@chatgpt-codex-connector chatgpt-codex-connector Bot Oct 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve fencing across rolling upgrades

During a rolling upgrade, old instances still use the legacy fence key while this head uses the newly namespaced key. If an old writer's distributed lock is lost early (for example, through independent Redis eviction or deletion) while its fence remains, a new instance can claim its different fence and complete a write, after which the delayed old writer can still validate against the legacy fence and overwrite the newer catalog. Keep the legacy and new versions on a shared fencing token/key, or add a migration protocol that checks both, so mixed-version writers continue to supersede one another.

AGENTS.md reference: AGENTS.md:L45-L48

Useful? React with 👍 / 👎.

@lia-by-librechat lia-by-librechat Bot Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 1c9c70b. New writers atomically claim the prefixed and exact legacy fence with one token/deadline, then validate the legacy fence when it is usable. Prefix-only ACLs fall back only after a pre-execution denial; other errors/conflicts abort. Added real Redis old→new and new→old tests with early lock deletion, stale-writer rejection, cleanup, and prefix-restricted ACL coverage. All 9 tests pass on both single-node Redis and a three-master cluster.


async function withRedisLock<T>(
lockKey: string,
Expand Down Expand Up @@ -431,7 +447,7 @@ export function createMCPCatalogStore(deps: CatalogStoreDeps): MCPCatalogStore {
const previous = userQueues.get(scope) ?? Promise.resolve();
const lockKey = `${CacheKeys.TOOL_CACHE}:tools:mcp-write-lock:${encodeURIComponent(userId)}:${encodeURIComponent(serverName)}`;
const generationRawKey = rawKey(ToolCacheKeys.MCP_SERVER_GENERATION(userId, serverName));
const fenceKey = `tools:mcp:write-fence:{${redisHashTag(generationRawKey)}}`;
const fenceKey = ownershipFenceKey('tools:mcp:write-fence', generationRawKey);
const locked = () =>
withRedisLock(lockKey, GLOBAL_LOCK_TTL_MS, (token, leaseExpiresAt) =>
requireOwnershipFence
Expand Down
Loading