Skip to content

perf: make dataloader doc placement O(1) with a key-index map#17469

Open
yoo-minho wants to merge 1 commit into
payloadcms:mainfrom
yoo-minho:perf/dataloader-o-n2-lookup
Open

perf: make dataloader doc placement O(1) with a key-index map#17469
yoo-minho wants to merge 1 commit into
payloadcms:mainfrom
yoo-minho:perf/dataloader-o-n2-lookup

Conversation

@yoo-minho

Copy link
Copy Markdown

What?

Replaces the per-doc keys.findIndex(...) scan in the collections dataloader with a single Map<string, number> lookup, turning the doc-placement loop from O(n²) into O(n) over the batch size.

Why?

batchLoadFn receives all cache keys requested in one tick (keys: readonly string[]). After the batched find returns, the current code places each doc back into the result array by scanning the entire keys array with findIndex, once per doc:

for (const doc of result.docs) {
  const docKey = createDataloaderCacheKey({ ... })
  const docsIndex = keys.findIndex((key) => key === docKey) // O(n) per doc
  if (docsIndex > -1) docs[docsIndex] = doc
}

Total docs across all batches ≈ keys.length, and each placement scans all of keys, so the work is O(keys.length²). keys grows with the number of documents loaded in a single tick — which, with depth-populated relations, can be hundreds or thousands — so this is quadratic on a hot batching path.

How?

keys is fixed for the batch, so I build a key → index map once up front and look up in O(1):

const keyToIndex = new Map<string, number>()
for (let i = 0; i < keys.length; i++) {
  if (!keyToIndex.has(keys[i]!)) keyToIndex.set(keys[i]!, i) // first occurrence wins, mirrors findIndex
}
// ...
const docsIndex = keyToIndex.get(docKey)
if (docsIndex !== undefined) docs[docsIndex] = doc

Behaviour is unchanged: findIndex returns the first match, and the map is built first-occurrence-wins to preserve that. Keys are unique per load in practice, so the guard is just defensive.

Targeting main (v4) as an enhancement, per the PR template.

Note: I couldn't run the full monorepo build/typecheck locally, but the change is minimal and type-checked against the existing signatures (keys: readonly string[], createDataloaderCacheKey(): string). Happy to adjust if CI surfaces anything.

Fixes #

The dataloader placed each returned doc by scanning the full keys array with
findIndex, once per doc — O(n²) over the batch size. Build a key->index Map once
(keys is fixed for the batch) and look up in O(1), making it O(n). First
occurrence wins, matching findIndex.
@yoo-minho yoo-minho changed the title perf(payload): O(1) dataloader doc placement via key→index Map perf: make dataloader doc placement O(1) with a key-index map Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant