Fix: mq.poll() materialized the entire queue to return one batch

[josh-j]

Problem

DataStoreMQ.poll() retrieves a batch of messages via ds.luaQuery(prefix, { limit }). However, queryLua materializes the entire prefix range from IndexedDB into memory first, and only then applies the limit via applyQuery:

// query_collection.ts
for await (let { key, value } of kv.query({ prefix })) {
  results.push(value);          // reads EVERY queued message
}
return applyQuery(results, query, env, sf, config);  // limit applied here

So polling 3 messages off a queue with N messages costs a full O(N) IndexedDB cursor walk. During a full space reindex the index queue holds every file in the space, and the queue is re-polled after each small batch — making total queue overhead O(N²/batchSize). On a ~4,000-file space that's millions of wasted cursor steps, a substantial share of total reindex wall-clock time, growing quadratically with space size.

Fix

Replace the luaQuery call with a direct cursor scan over ds.query({ prefix }) that breaks after maxItems. The IndexedDB-backed query() is a true cursor generator, so breaking early stops cursor advancement. Each poll is now O(batchSize) regardless of queue depth.

No behavior change otherwise: the same messages are returned in the same (key) order, and the move-to-processing logic is untouched.

Testing

• npx vitest run — 873 passed, no regressions (the handful of suite files that fail to load do so identically on main in my environment).
• Deployed on my own instance: full-space reindex of ~1,900 files shows the index queue no longer dominates profile time; behavior verified end-to-end (initial index completes, progress UI works, queue drains).

🤖 Generated with Claude Code