Skip to content

Commit afbb318

Browse files
committed
Add namespacePrefix option to share one component across aggregates
1 parent 243ccaf commit afbb318

6 files changed

Lines changed: 395 additions & 54 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,10 @@
11
# Changelog
22

3+
## Unreleased
4+
5+
- Support multiple independent aggregates in one component instance: pass
6+
`namespacePrefix` when constructing a `TableAggregate` or `DirectAggregate`.
7+
38
## 0.3.3
49

510
- Bump `convex-helpers` to 0.1.128, which isn’t affected by

‎README.md‎

Lines changed: 67 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,8 @@ const leaderboardByGame = new TableAggregate<{
129129
Key: number;
130130
DataModel: DataModel;
131131
TableName: "scores";
132-
}>(components.leaderboardByGame, {
132+
}>(components.aggregate, {
133+
namespacePrefix: "leaderboardByGame",
133134
namespace: (doc) => doc.gameId,
134135
sortKey: (doc) => doc.score,
135136
});
@@ -190,22 +191,6 @@ app.use(aggregate);
190191
export default app;
191192
```
192193

193-
### Defining multiple aggregates
194-
195-
You can aggregate multiple tables, multiple sort keys, or multiple values, but
196-
you need to make an instance of the aggregate component for each.
197-
198-
You do this by using the `aggregate` component multiple times, giving each usage
199-
its own name.
200-
201-
```ts
202-
app.use(aggregate, { name: "aggregateScores" });
203-
app.use(aggregate, { name: "aggregateByGame" });
204-
```
205-
206-
You then use the named aggregate when initializing the `TableAggregate` as we'll
207-
see below, using `components.aggregateScores` instead of `components.aggregate`.
208-
209194
## Usage
210195

211196
### Write to the aggregate data structure
@@ -228,6 +213,7 @@ const aggregate = new TableAggregate<{
228213
DataModel: DataModel;
229214
TableName: "mytable";
230215
}>(components.aggregate, {
216+
namespacePrefix: "mytableByCreationTime",
231217
sortKey: (doc) => doc._creationTime, // Allows querying across time ranges.
232218
sumValue: (doc) => doc.value, // The value to be used in `.sum` calculations.
233219
});
@@ -243,7 +229,8 @@ const aggregateByGame = new TableAggregate<{
243229
Key: [string, number];
244230
DataModel: DataModel;
245231
TableName: "leaderboard";
246-
}>(components.aggregateByGame, {
232+
}>(components.aggregate, {
233+
namespacePrefix: "byGame",
247234
namespace: (doc) => doc.gameId,
248235
sortKey: (doc) => [doc.username, doc.score],
249236
});
@@ -321,7 +308,10 @@ const randomize = new TableAggregate<{
321308
Key: null;
322309
DataModel: DataModel;
323310
TableName: "mytable";
324-
}>(components.aggregate, { sortKey: (doc) => null });
311+
}>(components.aggregate, {
312+
namespacePrefix: "randomize",
313+
sortKey: (doc) => null,
314+
});
325315
```
326316

327317
Without sorting, all documents are ordered by their `_id` which is generally
@@ -353,16 +343,14 @@ defineSchema({
353343
And an aggregate defined with key as `_creationTime` and namespace as `album`.
354344

355345
```ts
356-
// convex/convex.config.ts
357-
app.use(aggregate, { name: "photos" });
358-
359346
// convex/photos.ts
360347
const photos = new TableAggregate<{
361348
Namespace: string; // album name
362349
Key: number; // creation time
363350
DataModel: DataModel;
364351
TableName: "photos";
365-
}>(components.photos, {
352+
}>(components.aggregate, {
353+
namespacePrefix: "photos",
366354
namespace: (doc) => doc.album,
367355
sortKey: (doc) => doc._creationTime,
368356
});
@@ -404,6 +392,7 @@ import { DirectAggregate } from "@convex-dev/aggregate";
404392
// have the same key.
405393
const aggregate = new DirectAggregate<{ Key: number; Id: string }>(
406394
components.aggregate,
395+
{ namespacePrefix: "stats" },
407396
);
408397

409398
// within a mutation, add values to be aggregated
@@ -418,6 +407,40 @@ await aggregate.replace(ctx, { key: oldKey, id }, { key: newKey });
418407

419408
See [`example/convex/stats.ts`](example/convex/stats.ts) for an example.
420409

410+
### Defining multiple aggregates
411+
412+
You can aggregate multiple tables, multiple sort keys, or multiple values by
413+
defining multiple aggregates in one component, each with its own
414+
`namespacePrefix`. This works for both `TableAggregate` and `DirectAggregate`.
415+
416+
```ts
417+
const aggregateScores = new TableAggregate<...>(components.aggregate, {
418+
namespacePrefix: "scores",
419+
sortKey: (doc) => doc.score,
420+
});
421+
const aggregateByGame = new TableAggregate<...>(components.aggregate, {
422+
namespacePrefix: "byGame",
423+
namespace: (doc) => doc.gameId,
424+
sortKey: (doc) => doc.score,
425+
});
426+
const latencies = new DirectAggregate<{ Key: number; Id: string }>(
427+
components.aggregate,
428+
{ namespacePrefix: "latencies" },
429+
);
430+
```
431+
432+
The aggregate stores each namespace `ns` as `[namespacePrefix, ns]`, and data
433+
without a namespace as `[namespacePrefix]`, so each aggregate gets its own
434+
internal data structures. You keep passing the namespace without the prefix,
435+
e.g. `aggregateByGame.count(ctx, { namespace: gameId })`, and `iterNamespaces`
436+
and `clearAll` only cover the aggregate's own namespaces.
437+
438+
The `namespacePrefix` determines where an aggregate's data is stored, so
439+
changing it points the aggregate at a new, empty one. Aggregates defined without
440+
a `namespacePrefix` store their data under the namespaces you give them
441+
directly, and their `iterNamespaces` and `clearAll` cover every namespace in the
442+
component, including those of prefixed aggregates.
443+
421444
## Operations
422445

423446
You've set up your aggregate. Now let's see how to backfill it to account for
@@ -496,9 +519,9 @@ data without writing to the aggregate, they can get out of sync and the returned
496519
aggregates may be incorrect.
497520

498521
The simplest way to fix is to start over. Either call
499-
`await aggregate.clear(ctx)` or rename the component like
500-
`app.use(aggregate, { name: "newName" })` which will reset it to be empty. Then
501-
follow the instructions from [above](#attach-aggregate-to-an-existing-table).
522+
`await aggregate.clearAll(ctx)` or change its `namespacePrefix` to a new prefix,
523+
which starts out empty. Then follow the instructions from
524+
[above](#attach-aggregate-to-an-existing-table).
502525

503526
There is an alternative which doesn't clear the aggregates: compare the source
504527
of truth to the aggregate table. You can use `db.query("mytable").paginate()` on
@@ -755,7 +778,24 @@ reads are still reactive to changes to the aggregate as the Batch Worker
755778
processes the queued writes.
756779

757780
Note that you cannot mix the queued mode with the non-queued mode. If there are
758-
any writes queued, a non-stale read or non-async write will throw an error.
781+
any writes queued, a non-stale read or non-async write will throw an error. This
782+
applies across all aggregates in the component, so either all or none of the
783+
aggregates in a component should use `{ async: true }` writes and
784+
`{ stale: true }` reads. If you need both modes, install the component a second
785+
time under its own name, and use that instance for the queued aggregates:
786+
787+
```ts
788+
// convex/convex.config.ts
789+
app.use(aggregate);
790+
app.use(aggregate, { name: "queuedAggregate" });
791+
```
792+
793+
```ts
794+
const clicks = new TableAggregate<...>(components.queuedAggregate, {
795+
namespacePrefix: "clicks",
796+
sortKey: (doc) => doc._creationTime,
797+
});
798+
```
759799

760800
#### Dead-lettered writes
761801

‎src/client/index.test.ts‎

Lines changed: 201 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
import { afterEach, beforeEach, describe, expect, test, vi } from "vitest";
2-
import { TableAggregate } from "./index.js";
2+
import { DirectAggregate, TableAggregate } from "./index.js";
33
import {
44
components,
55
componentSchema,
@@ -13,6 +13,7 @@ import {
1313
} from "convex/server";
1414
import { v } from "convex/values";
1515
import { convexTest } from "convex-test";
16+
import batchWorker from "@convex-dev/batch-worker/test";
1617
import type { DataModelFromSchemaDefinition } from "convex/server";
1718

1819
const schema = defineSchema({
@@ -1236,3 +1237,202 @@ describe("stale reads", () => {
12361237
expect(count).toBe(1);
12371238
});
12381239
});
1240+
1241+
describe("namespacePrefix", () => {
1242+
function createPrefixedAggregates() {
1243+
const unprefixed = new TableAggregate<{
1244+
Key: number;
1245+
DataModel: DataModel;
1246+
TableName: "testItems";
1247+
}>(components.aggregate, {
1248+
sortKey: (doc) => doc.value,
1249+
});
1250+
const byValue = new TableAggregate<{
1251+
Key: number;
1252+
DataModel: DataModel;
1253+
TableName: "testItems";
1254+
}>(components.aggregate, {
1255+
namespacePrefix: "byValue",
1256+
sortKey: (doc) => doc.value,
1257+
sumValue: (doc) => doc.value,
1258+
});
1259+
const byName = new TableAggregate<{
1260+
Key: string;
1261+
DataModel: DataModel;
1262+
TableName: "testItems";
1263+
}>(components.aggregate, {
1264+
namespacePrefix: "byName",
1265+
sortKey: (doc) => doc.name,
1266+
});
1267+
const byAlbum = new TableAggregate<{
1268+
Namespace: string;
1269+
Key: number;
1270+
DataModel: DataModel;
1271+
TableName: "photos";
1272+
}>(components.aggregate, {
1273+
namespacePrefix: "byAlbum",
1274+
namespace: (doc) => doc.album,
1275+
sortKey: (doc) => doc.score,
1276+
});
1277+
return { unprefixed, byValue, byName, byAlbum };
1278+
}
1279+
1280+
async function insertPhoto(
1281+
ctx: GenericMutationCtx<DataModel>,
1282+
album: string,
1283+
score: number,
1284+
) {
1285+
const id = await ctx.db.insert("photos", { album, url: "u", score });
1286+
return (await ctx.db.get("photos", id))!;
1287+
}
1288+
1289+
test("prefixed aggregates are independent of each other", async () => {
1290+
const t = setupTest();
1291+
const { unprefixed, byValue, byName } = createPrefixedAggregates();
1292+
1293+
await t.run(async (ctx) => {
1294+
const a = await testItem(ctx, { name: "a", value: 1 });
1295+
const b = await testItem(ctx, { name: "b", value: 2 });
1296+
await byValue.insert(ctx, a);
1297+
await byValue.insert(ctx, b);
1298+
await byName.insert(ctx, a);
1299+
await unprefixed.insert(ctx, a);
1300+
await unprefixed.insert(ctx, b);
1301+
await unprefixed.insert(
1302+
ctx,
1303+
await testItem(ctx, { name: "c", value: 3 }),
1304+
);
1305+
});
1306+
1307+
await t.run(async (ctx) => {
1308+
expect(await byValue.count(ctx)).toBe(2);
1309+
expect(await byValue.sum(ctx)).toBe(3);
1310+
expect(await byName.count(ctx)).toBe(1);
1311+
expect((await byName.max(ctx))?.key).toBe("a");
1312+
expect(await unprefixed.count(ctx)).toBe(3);
1313+
});
1314+
});
1315+
1316+
test("namespaces are scoped to the prefix", async () => {
1317+
const t = setupTest();
1318+
const { byAlbum } = createPrefixedAggregates();
1319+
const otherAlbums = new TableAggregate<{
1320+
Namespace: string;
1321+
Key: number;
1322+
DataModel: DataModel;
1323+
TableName: "photos";
1324+
}>(components.aggregate, {
1325+
namespacePrefix: "byAlbum2",
1326+
namespace: (doc) => doc.album,
1327+
sortKey: (doc) => doc.score,
1328+
});
1329+
1330+
await t.run(async (ctx) => {
1331+
await byAlbum.insert(ctx, await insertPhoto(ctx, "x", 1));
1332+
await byAlbum.insert(ctx, await insertPhoto(ctx, "x", 2));
1333+
await byAlbum.insert(ctx, await insertPhoto(ctx, "y", 3));
1334+
await byAlbum.insert(ctx, await insertPhoto(ctx, "z", 4));
1335+
await otherAlbums.insert(ctx, await insertPhoto(ctx, "x", 5));
1336+
});
1337+
1338+
await t.run(async (ctx) => {
1339+
expect(await byAlbum.count(ctx, { namespace: "x" })).toBe(2);
1340+
expect(await otherAlbums.count(ctx, { namespace: "x" })).toBe(1);
1341+
const namespaces = [];
1342+
for await (const ns of byAlbum.iterNamespaces(ctx, 1)) {
1343+
namespaces.push(ns);
1344+
}
1345+
expect(namespaces).toEqual(["x", "y", "z"]);
1346+
const other = [];
1347+
for await (const ns of otherAlbums.iterNamespaces(ctx)) {
1348+
other.push(ns);
1349+
}
1350+
expect(other).toEqual(["x"]);
1351+
});
1352+
});
1353+
1354+
test("clearAll only clears the prefixed aggregate", async () => {
1355+
const t = setupTest();
1356+
const { unprefixed, byValue, byAlbum } = createPrefixedAggregates();
1357+
1358+
await t.run(async (ctx) => {
1359+
const a = await testItem(ctx, { name: "a", value: 1 });
1360+
await byValue.insert(ctx, a);
1361+
await unprefixed.insert(ctx, a);
1362+
await byAlbum.insert(ctx, await insertPhoto(ctx, "x", 1));
1363+
});
1364+
1365+
await t.run(async (ctx) => {
1366+
await byAlbum.clearAll(ctx);
1367+
expect(await byAlbum.count(ctx, { namespace: "x" })).toBe(0);
1368+
expect(await byValue.count(ctx)).toBe(1);
1369+
expect(await unprefixed.count(ctx)).toBe(1);
1370+
});
1371+
});
1372+
1373+
test("DirectAggregate supports a namespacePrefix", async () => {
1374+
const t = setupTest();
1375+
const direct = new DirectAggregate<{ Key: number; Id: string }>(
1376+
components.aggregate,
1377+
{ namespacePrefix: "direct" },
1378+
);
1379+
const other = new DirectAggregate<{ Key: number; Id: string }>(
1380+
components.aggregate,
1381+
);
1382+
1383+
await t.run(async (ctx) => {
1384+
await direct.insert(ctx, { key: 1, id: "a", sumValue: 10 });
1385+
await direct.insert(ctx, { key: 2, id: "b", sumValue: 20 });
1386+
await direct.replace(ctx, { key: 2, id: "b" }, { key: 3, sumValue: 30 });
1387+
await other.insert(ctx, { key: 1, id: "a" });
1388+
});
1389+
1390+
await t.run(async (ctx) => {
1391+
expect(await direct.count(ctx)).toBe(2);
1392+
expect(await direct.sum(ctx)).toBe(40);
1393+
expect(
1394+
await direct.countBatch(ctx, [
1395+
{},
1396+
{ bounds: { lower: { key: 2, inclusive: true } } },
1397+
]),
1398+
).toEqual([2, 1]);
1399+
expect(await other.count(ctx)).toBe(1);
1400+
});
1401+
});
1402+
1403+
test("queued writes use the prefix", async () => {
1404+
vi.useFakeTimers();
1405+
try {
1406+
const t = setupTest();
1407+
batchWorker.register(t, "aggregate/batchWorker");
1408+
const direct = new DirectAggregate<{ Key: number; Id: string }>(
1409+
components.aggregate,
1410+
{ namespacePrefix: "direct" },
1411+
);
1412+
const other = new DirectAggregate<{ Key: number; Id: string }>(
1413+
components.aggregate,
1414+
);
1415+
1416+
await t.run(async (ctx) => {
1417+
await direct.insert(ctx, { key: 1, id: "a" }, { async: true });
1418+
await direct.enqueueBatch(ctx, [
1419+
{ type: "insert", key: 2, id: "b" },
1420+
{
1421+
type: "replace",
1422+
currentItem: { key: 1, id: "a" },
1423+
newItem: { key: 4 },
1424+
},
1425+
]);
1426+
});
1427+
await t.finishAllScheduledFunctions(vi.runAllTimers);
1428+
1429+
await t.run(async (ctx) => {
1430+
expect(await direct.count(ctx)).toBe(2);
1431+
expect((await direct.max(ctx))?.key).toBe(4);
1432+
expect(await other.count(ctx)).toBe(0);
1433+
});
1434+
} finally {
1435+
vi.useRealTimers();
1436+
}
1437+
});
1438+
});

0 commit comments

Comments
 (0)