Postgres (#5365)

* postgres experimental WIP

* astonishingly, all mocha tests of apostrophe pass with this

* mocha tests pass, actual sites work

* lint clean

* listDatabases support, but changes are coming

* wip

* dump and restore updates

* backpressure, adequate handling of ObjectId for our needs (becomes its hex representation)

* mild performance optimization

* profiling

* testing issue resolved

* refactored to db-connect module, introduced sqlite adapter

* sqlite WIP

* debugging

* programmatic API for dump/restore/copy dbs

* linting, documentation

* MIT license

* text ranking is more accurate, documentation is more complete

* good full text search for sqlite

* updates for compatibility with the rest of the public and private modules, plus a few fixes to genuinely ambiguous tests

* requirements found by testing private modules

* fixes from full cypress run

* eslint passing

* restore permissions

* maximize atomicity

* bug fixes

* * exit properly when asset tests fail
* "npm test" tests all three adapters

* ignore claude-tools in eslint

* postgres and sqlite-inclusive ci matrix attempt

* clean up logs

* We hit github's limit on total configurations because every package gets its own matrix.

Solve that with grouping:

* apostrophe core
* All regular ecosystem packages other than core
* non-database-requiring packges
* mongodb-specific packages

This will probably speed it up too because it won't have to spin up a container a bazillion times.

* hardened the asset tests, made them less timing sensitive, fixed a bad commit resulting from the
way they dodgily patch themselves without a robust cleanup mechanism

* fix a root cause of asset test instability

* log mess

* implemented missing $size operator

* test compatibility

* advanced permission uses regex in $in

* regex in $in

* .db() should not make false promises in plain postgres mode, it should fail

* ability to specify a default adapter

* obsolete file

* put escapeHost back where it belongs

* dead code removal, test cleanup

* emulate-mongo-3-driver only needed in db-connect

* no claude logs in repo (tools are welcome)

* * shared aggregation implementation, other shared things
* optimize $match when it is the first step in aggregation, don't fetch the whole collection 😜
* multipostgres listDatabases() and .db() should return and expect "fully qualified virtual database names," e.g. physical_db_name-schemaname

* vanilla postgres should not attempt to use .db() with alternate names in tests

* documentation corrections

* documentation errors

* listDatabases and documentation corrections

* more edge cases revealed by latest work from Miro

* anchored prefix regexps are optimized
documentation improvements

* * matchesQuery in the aggregation cursor implementation doesn't throw on unrecognized operators. It should, and it should support the same mongodb operators that the regular find()
  path does in postgres/sqlite (our official subset), unless there is an extraordinary reason not to.

  * Similarly, the main query implementation for normal queries should throw on unrecognized operators if it doesn't already.

  * The dump/restore programmatic APIs in db-connect concern me. These involve returning the entire database as a string, which could exhaust memory. This impacts both utilities and
  also copyDatabase(). Could these APIs return and expect async iterators instead of strings?

  * The test "anchored regex on an indexed field uses a btree index search" runs explain on a query that's hardcoded in the test. Instead these SQL based adapters should expose a means
  to get the SQL for a query, so it can be directly tested. Otherwise this test proves nothing as changes to the adapter accumulate in future.

  * Why is this test searching for "at least 1" and not exactly 1?

  it('should find documents with null value', async function() {
    const docs = await db.collection('test').find({ value: null }).toArray();
    // MongoDB matches both null and missing fields with { value: null }
    expect(docs.length).to.be.at.least(1);
  });

  * What is the maximum size of a db-connect document in the postgres and sqlite adapters?

  * Update the copyright year in db-connect/LICENSE.md to 2025.

  * The db-connect README mentions: sqlite://:memory: What happens if you try to use .db('some-name') with that? I think it would be best to just not support throwaway in-memory sqlite
  databases because I doubt anyone would intentionally store a website in one.

* do not swallow dump/restore errors on indexes

* cover how to run the utilities

* fix detection of source

* separate sanitization for index names

* regex prefix safety

* pnpm

---------

Co-authored-by: Thomas Boutell <boutell@vcs.trox.local>

Author: boutell

.github/workflows/monorepo.yml

@@ -7,6 +7,7 @@ env:
   NODE_VERSIONS_JSON: "[20,22,24]"
   MONGODB_VERSIONS_JSON: '["7","8"]'
   REDIS_VERSION: "7"
+  POSTGRES_VERSION: "16"
 
 on:
   push:
@@ -124,7 +125,7 @@ jobs:
         run: |
           month='${{ steps.cache-month.outputs.value }}'
           mongo_suffix=$(jq -r '.[]' <<< '${{ env.MONGODB_VERSIONS_JSON }}' | paste -sd'-' -)
-          echo "key=docker-images-${month}-mongo${mongo_suffix}-redis${{ env.REDIS_VERSION }}" >> "$GITHUB_OUTPUT"
+          echo "key=docker-images-${month}-mongo${mongo_suffix}-redis${{ env.REDIS_VERSION }}-pg${{ env.POSTGRES_VERSION }}" >> "$GITHUB_OUTPUT"
 
       - name: Restore docker image cache
         id: docker-cache
@@ -143,6 +144,8 @@ jobs:
           done
           docker pull "redis:${{ env.REDIS_VERSION }}"
           docker save "redis:${{ env.REDIS_VERSION }}" -o ".github/docker-cache/redis-${{ env.REDIS_VERSION }}.tar"
+          docker pull "postgres:${{ env.POSTGRES_VERSION }}"
+          docker save "postgres:${{ env.POSTGRES_VERSION }}" -o ".github/docker-cache/postgres-${{ env.POSTGRES_VERSION }}.tar"
 
   warm-sharp-cache:
     name: Warm sharp/libvips cache (Node ${{ matrix.nodeVersion }})
@@ -216,7 +219,7 @@ jobs:
           du -sh ~/.npm/_libvips || true
 
   package-tests:
-    name: ${{ format('{0} ({1}, {2})', matrix.package, matrix.nodeVersion, matrix.needsMongo && matrix.mongodbVersion || 'n/a') }}
+    name: ${{ format('{0} ({1}, {2}{3})', matrix.group, matrix.nodeVersion, matrix.adapter || 'n/a', matrix.needsMongo && format(' mongo {0}', matrix.mongodbVersion) || '') }}
     needs:
       - setup
       - shared-runtime
@@ -279,7 +282,7 @@ jobs:
       - name: Restore docker image cache
         id: docker-cache-restore
         uses: actions/cache/restore@v4
-        if: matrix.needsMongo || matrix.needsRedis
+        if: matrix.needsMongo || matrix.needsRedis || matrix.needsPostgres
         with:
           path: .github/docker-cache
           key: ${{ needs.shared-runtime.outputs.docker-cache-key }}
@@ -300,6 +303,14 @@ jobs:
         if: steps.docker-cache-restore.outputs.cache-hit != 'true' && matrix.needsRedis
         run: docker pull redis:${{ env.REDIS_VERSION }}
 
+      - name: Load cached Postgres image
+        if: steps.docker-cache-restore.outputs.cache-hit == 'true' && matrix.needsPostgres
+        run: docker load -i ".github/docker-cache/postgres-${{ env.POSTGRES_VERSION }}.tar"
+
+      - name: Pull Postgres image (cache miss)
+        if: steps.docker-cache-restore.outputs.cache-hit != 'true' && matrix.needsPostgres
+        run: docker pull postgres:${{ env.POSTGRES_VERSION }}
+
       - name: Start MongoDB
         if: matrix.needsMongo
         run: |
@@ -358,15 +369,64 @@ jobs:
           docker logs redis
           exit 1
 
+      - name: Start PostgreSQL
+        if: matrix.needsPostgres
+        run: |
+          docker rm -f postgres >/dev/null 2>&1 || true
+          docker run -d \
+            --name postgres \
+            --publish 5432:5432 \
+            -e POSTGRES_HOST_AUTH_METHOD=trust \
+            --health-cmd "pg_isready -U postgres" \
+            --health-interval 5s \
+            --health-timeout 5s \
+            --health-retries 12 \
+            postgres:${{ env.POSTGRES_VERSION }}
+          echo "Waiting for PostgreSQL to report healthy..."
+          for attempt in $(seq 1 60); do
+            status=$(docker inspect --format='{{.State.Health.Status}}' postgres 2>/dev/null || echo "starting")
+            if [ "$status" = "healthy" ]; then
+              exit 0
+            fi
+            if [ "$status" = "unhealthy" ]; then
+              echo "PostgreSQL reported unhealthy" >&2
+              docker logs postgres
+              exit 1
+            fi
+            sleep 2
+          done
+          echo "PostgreSQL failed to become healthy in time" >&2
+          docker logs postgres
+          exit 1
+
       - name: Install workspace dependencies
         run: pnpm install --frozen-lockfile
 
       - name: Run package tests
-        run: pnpm run --filter "${{ matrix.package }}" --if-present test
+        run: |
+          failed=0
+          for pkg in $(echo '${{ matrix.packages }}' | jq -r '.[]'); do
+            echo "::group::Testing $pkg"
+            if ! pnpm run --filter "$pkg" --if-present test; then
+              echo "::error::Tests failed for $pkg"
+              failed=1
+            fi
+            echo "::endgroup::"
+          done
+          exit $failed
         env:
           CI: true
           # Yes we want import-export to test with automatic-translation
           TEST_WITH_PRO: "1"
+          # Adapter selection: mongodb (default), postgres, or sqlite.
+          # ADAPTER is used by db-connect tests, APOS_TEST_DB_PROTOCOL by apostrophe tests.
+          ADAPTER: ${{ matrix.adapter }}
+          APOS_TEST_DB_PROTOCOL: ${{ matrix.adapter }}
+          PGUSER: postgres
+
+      - name: Stop PostgreSQL
+        if: always() && matrix.needsPostgres
+        run: docker rm -f postgres || true
 
       - name: Stop Redis
         if: always() && matrix.needsRedis

.github/workflows/scripts/detect-impacted-packages.mjs

@@ -73,7 +73,8 @@ async function main() {
       package: name,
       directory: packages.get(name).relativeDir,
       requiresMongo: packages.get(name).requiresMongo !== false,
-      requiresRedis: packages.get(name).requiresRedis === true
+      requiresRedis: packages.get(name).requiresRedis === true,
+      mongodbOnly: packages.get(name).mongodbOnly === true
     }))
   };
 
@@ -130,14 +131,16 @@ async function loadPackages() {
       const testConfig = manifest.apostropheTestConfig || {};
       const requiresMongo = testConfig.requiresMongo !== false;
       const requiresRedis = testConfig.requiresRedis === true;
+      const mongodbOnly = testConfig.mongodbOnly === true;
 
       map.set(manifest.name, {
         name: manifest.name,
         relativeDir: path.posix.join('packages', entry.name),
         dependencies,
         hasTestScript,
         requiresMongo,
-        requiresRedis
+        requiresRedis,
+        mongodbOnly
       });
     }));
 

.github/workflows/scripts/expand-runtime-matrix.mjs

@@ -1,6 +1,21 @@
 #!/usr/bin/env node
-// Expands the impacted package matrix with runtime permutations supplied 
+// Expands the impacted package matrix with runtime permutations supplied
 // via env vars.
+//
+// Packages are grouped into jobs to stay within GitHub's 256-entry matrix limit:
+//   - "apostrophe" runs solo (the main package, benefits from its own status).
+//   - All other database packages are grouped into an "ecosystem" job.
+//   - mongodbOnly packages are grouped into "ecosystem-mongodb".
+//   - Non-database packages are grouped into "standalone".
+//
+// For groups that need a database, three adapter variants are emitted:
+//   1. mongodb  – all Node versions × all MongoDB versions
+//   2. postgres – latest LTS Node only, no MongoDB
+//   3. sqlite   – latest LTS Node only, no MongoDB
+//
+// The latest LTS Node version is the highest even-numbered entry in
+// NODE_VERSIONS_JSON.
+
 import { readFile } from 'fs/promises';
 
 const args = process.argv.slice(2);
@@ -36,25 +51,120 @@ function parseJsonArray(name, raw) {
 const nodeVersions = parseJsonArray('NODE_VERSIONS_JSON', process.env.NODE_VERSIONS_JSON);
 const mongodbVersions = parseJsonArray('MONGODB_VERSIONS_JSON', process.env.MONGODB_VERSIONS_JSON);
 
+// Latest LTS = highest even-numbered Node version
+const latestLts = [...nodeVersions]
+  .filter((v) => Number(v) % 2 === 0)
+  .sort((a, b) => Number(b) - Number(a))[0];
+
 const impact = JSON.parse(await readFile(impactPath, 'utf8'));
 const packages = impact?.matrix?.include || [];
-const include = [];
+
+// The main apostrophe package always gets its own jobs for clear CI status.
+const SOLO_PACKAGES = new Set(['apostrophe']);
+
+// Sort packages into groups
+const solo = [];
+const ecosystem = [];
+const ecosystemMongodbOnly = [];
+const standalone = [];
 
 for (const pkg of packages) {
-  const needsMongo = pkg.requiresMongo !== false;
-  const needsRedis = pkg.requiresRedis === true;
-  const mongoTargets = needsMongo ? mongodbVersions : [''];
+  const needsDb = pkg.requiresMongo !== false;
+  if (SOLO_PACKAGES.has(pkg.package)) {
+    solo.push(pkg);
+  } else if (needsDb && pkg.mongodbOnly) {
+    ecosystemMongodbOnly.push(pkg);
+  } else if (needsDb) {
+    ecosystem.push(pkg);
+  } else {
+    standalone.push(pkg);
+  }
+}
+
+const include = [];
+
+// Emit runtime combinations for a group of packages.
+function emitGroup(group, pkgs) {
+  if (!pkgs.length) {
+    return;
+  }
+  const packageNames = JSON.stringify(pkgs.map((p) => p.package));
+  const needsRedis = pkgs.some((p) => p.requiresRedis === true);
+  const mongodbOnly = pkgs.every((p) => p.mongodbOnly);
+
+  // mongodb: all Node versions × all MongoDB versions
   for (const nodeVersion of nodeVersions) {
-    for (const mongodbVersion of mongoTargets) {
+    for (const mongodbVersion of mongodbVersions) {
       include.push({
-        ...pkg,
+        group,
+        packages: packageNames,
         nodeVersion,
         mongodbVersion,
-        needsMongo,
+        adapter: 'mongodb',
+        needsMongo: true,
+        needsPostgres: false,
         needsRedis
       });
     }
   }
+  // postgres and sqlite: latest LTS only, skip for mongodb-only groups
+  if (!mongodbOnly) {
+    include.push({
+      group,
+      packages: packageNames,
+      nodeVersion: latestLts,
+      mongodbVersion: '',
+      adapter: 'postgres',
+      needsMongo: false,
+      needsPostgres: true,
+      needsRedis
+    });
+    include.push({
+      group,
+      packages: packageNames,
+      nodeVersion: latestLts,
+      mongodbVersion: '',
+      adapter: 'sqlite',
+      needsMongo: false,
+      needsPostgres: false,
+      needsRedis
+    });
+  }
+}
+
+// Emit non-database group (no adapter variants, just Node versions)
+function emitStandalone(group, pkgs) {
+  if (!pkgs.length) {
+    return;
+  }
+  const packageNames = JSON.stringify(pkgs.map((p) => p.package));
+  for (const nodeVersion of nodeVersions) {
+    include.push({
+      group,
+      packages: packageNames,
+      nodeVersion,
+      mongodbVersion: '',
+      adapter: '',
+      needsMongo: false,
+      needsPostgres: false,
+      needsRedis: false
+    });
+  }
+}
+
+// Solo packages each get their own group
+for (const pkg of solo) {
+  emitGroup(pkg.package, [pkg]);
+}
+
+if (ecosystem.length) {
+  emitGroup('ecosystem', ecosystem);
+}
+if (ecosystemMongodbOnly.length) {
+  emitGroup('ecosystem-mongodb', ecosystemMongodbOnly);
+}
+if (standalone.length) {
+  emitStandalone('standalone', standalone);
 }
 
 process.stdout.write(JSON.stringify({ include }));

.gitignore

@@ -6,3 +6,5 @@ public/apos-frontend
 .DS_Store
 coverage/
 .nyc_output
+claude-tools/logs/
+.claude

claude-tools/run-core-tests.sh

@@ -0,0 +1,37 @@
+#!/bin/bash
+# Run the apostrophe core test suite against a chosen DB adapter and log
+# output to claude-tools/logs/core-<adapter>.log. Usage:
+#
+#   ./claude-tools/run-core-tests.sh mongodb
+#   ./claude-tools/run-core-tests.sh postgres
+#   ./claude-tools/run-core-tests.sh sqlite
+#
+# NEVER run multiple adapters in parallel — the test suite is not designed
+# for concurrent runs and the host has limited resources.
+
+set -u
+adapter="${1:-}"
+if [[ -z "$adapter" ]]; then
+  echo "usage: $0 <mongodb|postgres|sqlite>" >&2
+  exit 2
+fi
+
+root="$(cd "$(dirname "$0")/.." && pwd)"
+logdir="$root/claude-tools/logs"
+mkdir -p "$logdir"
+log="$logdir/core-$adapter.log"
+: > "$log"
+
+echo "=== $adapter core tests ($(date -Is)) ===" | tee -a "$log"
+
+cd "$root/packages/apostrophe"
+
+extra=()
+if [[ "$adapter" == "postgres" ]]; then
+  extra=(env PGPASSWORD=testpassword)
+fi
+
+APOS_TEST_DB_PROTOCOL="$adapter" "${extra[@]}" npm run test:base >> "$log" 2>&1
+code=$?
+echo "=== exit=$code ===" | tee -a "$log"
+exit "$code"

package.json

@@ -5,7 +5,8 @@
     "dev": "pnpm --parallel --recursive run dev",
     "build": "pnpm --recursive run build",
     "lint": "pnpm --recursive run lint",
-    "test": "pnpm --recursive run test",
+    "test": "APOS_TEST_DB_PROTOCOL=postgres pnpm run test:main && APOS_TEST_DB_PROTOCOL=mongodb pnpm run test:main && APOS_TEST_DB_PROTOCOL=sqlite pnpm run test:main && APOS_TEST_DB_PROTOCOL=multipostgres pnpm run test:main",
+    "test:main": "echo \"APOS_TEST_DB_PROTOCOL IS: $APOS_TEST_DB_PROTOCOL\" && pnpm --recursive run test",
     "eslint": "pnpm --recursive run eslint",
     "mocha": "pnpm --recursive run mocha",
     "clean": "pnpm -r exec rm -rf node_modules && rm -rf node_modules && rm pnpm-lock.yaml"
@@ -18,6 +19,7 @@
   },
   "pnpm": {
     "onlyBuiltDependencies": [
+      "better-sqlite3",
       "sharp",
       "vue-demi",
       "@parcel/watcher",

packages/apostrophe/.gitignore

@@ -42,3 +42,6 @@ test/public/uploads
 
 # vim swp files
 .*.sw*
+
+# claude-tools log files
+claude-tools/**/*.log