Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 11 additions & 39 deletions README-MIGRATIONS.md
Original file line number Diff line number Diff line change
@@ -1,44 +1,16 @@
# Managing Database Migrations
# Database Migrations

This project uses a custom raw SQL migration script to manage database schema changes reliably.
## Prefix Numbering Rule

## Structure

- `src/db/migrations/`: Directory where all `.sql` migration files are stored.
- `src/db/migrate.ts`: The script that executes pending migrations against the database.
All database migrations must follow a strict prefix numbering convention to ensure deterministic execution order and prevent collisions.

## Creating Migrations
### Rules

To add a new migration, create a new `.sql` file in `src/db/migrations/`.
1. **Numeric Prefix**: Every migration file MUST begin with a numeric prefix followed by an underscore (e.g., `001_create_users.sql`).
2. **Sequential Ordering**: The prefixes MUST be strictly monotonic and strictly increasing.
3. **No Duplicates**: Duplicate prefixes are NOT allowed. If two developers create a migration at the same time with the same prefix, one must be renamed during the merge process.
4. **No Out-of-Band Migrations**: The `999_*` prefix is flagged as an out-of-band prefix and is rejected by the system.
5. **Extension**: All migration files MUST have the `.sql` extension.
6. **Hidden Files**: Hidden files (starting with `.`) are ignored.

**Naming Convention & Safety Rules:**
Use a sequential, zero-padded numeric prefix followed by a descriptive name: `XXX_description.sql` (e.g., `003_add_user_status.sql`).

To ensure safety and execution predictability, the following strict rules are enforced:
1. **Unique Prefixes**: Every migration file must have a unique sequential prefix. Duplicate prefixes (e.g., two `001_*` files) are strictly rejected to prevent execution order ambiguity and collisions.
2. **Strict Monotonicity**: Alphabetic ordering of the filenames must match their numeric prefix sequence. Non-padded or incorrectly ordered prefixes that violate monotonic progression are caught and rejected.
3. **Out-of-band Prefix (`999_`)**: Filenames starting with `999_*` are flagged as "out-of-band" migrations (used for temporary/development purposes) and will be rejected unless explicit relaxed options are enabled.
4. **Valid File Extension**: Every migration file must end with `.sql`. Files with incorrect extensions (e.g., `.sql.bak` or `.txt`) will be rejected.
5. **Hidden Files**: Any hidden files starting with a dot (e.g., `.DS_Store` or `.gitkeep`) are automatically ignored during migration resolution.

## Running Migrations

Migrations rely on the `DATABASE_URL` environment variable.

1. Ensure your `.env` file has a valid `DATABASE_URL`:
```env
DATABASE_URL="postgres://user:password@localhost:5432/revora"
```
2. Run the migration script via npm:
```bash
npm run migrate
```

This command will:
1. Compile the TypeScript code (`tsc`).
2. Connect to the database specified by `DATABASE_URL`.
3. Create the `schema_version` table if it doesn't already exist.
4. Apply any `.sql` file in `src/db/migrations/` that hasn't been recorded in `schema_version`, within a transaction.
5. Record the applied filename in `schema_version`.

If a migration fails mid-execution, the transaction will rollback, leaving your database safely unmodified.
These rules are enforced by the `resolveMigrations` function in `src/db/migrate.js` during the test run and at startup. A duplicate numeric prefix or non-monotonic ordering will fail the test run with a clear diagnostic.
4 changes: 2 additions & 2 deletions src/db/migrate.js
Original file line number Diff line number Diff line change
Expand Up @@ -128,9 +128,9 @@ async function runMigrations() {
return;
}

// Load and resolve migrations with relaxed options for the pre-existing production folder
// Load and resolve migrations with strict options
const allFiles = fs.readdirSync(migrationsDir);
const files = resolveMigrations(allFiles, { allowDuplicates: true, allowOutOfBand: true, strictExtensions: false });
const files = resolveMigrations(allFiles);

let appliedCount = 0;

Expand Down
29 changes: 14 additions & 15 deletions src/db/migrate.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -121,28 +121,27 @@ describe('Database Migration Ordering and Collision Resolver', () => {
});

describe('On-Disk Active Migration List Assertions', () => {
it('should successfully validate the on-disk migrations directory under production rules', () => {
it('should successfully validate the on-disk migrations directory under strict rules', () => {
const migrationsDir = path.join(__dirname, 'migrations');
expect(fs.existsSync(migrationsDir)).toBe(true);

const allFiles = fs.readdirSync(migrationsDir);

// The on-disk list contains duplicates (001, 002, 011) and out-of-band prefix 999.
// Therefore, it must succeed when using the Option A fallback options.
const resolved = resolveMigrations(allFiles, { allowDuplicates: true, allowOutOfBand: true, strictExtensions: false });
// The on-disk list must pass strict validation with no duplicates or out-of-band prefixes.
const resolved = resolveMigrations(allFiles);

expect(resolved.length).toBeGreaterThan(0);
expect(resolved.some(f => f.startsWith('001_'))).toBe(true);
expect(resolved.some(f => f.startsWith('999_'))).toBe(true);
});

it('should fail with duplicate prefix error if duplicates are not allowed on the real on-disk files', () => {
const migrationsDir = path.join(__dirname, 'migrations');
const allFiles = fs.readdirSync(migrationsDir);

expect(() => {
resolveMigrations(allFiles, { allowDuplicates: false, allowOutOfBand: true, strictExtensions: false });
}).toThrow('Duplicate migration prefix detected');

let lastPrefixNum = -1;
for (const filename of resolved) {
const match = filename.match(/^(\d+)_.*\.sql$/);
expect(match).not.toBeNull();
if (match) {
const prefixNum = parseInt(match[1], 10);
expect(prefixNum).toBeGreaterThan(lastPrefixNum);
lastPrefixNum = prefixNum;
}
}
});
});
});
6 changes: 3 additions & 3 deletions src/db/migrations/safety/executor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -598,9 +598,9 @@ export class MigrationManager {
const appliedMigrations = await this.getAppliedMigrations();
const appliedSet = new Set(appliedMigrations);

const files = fs.readdirSync(migrationsDir)
.filter((f: string) => f.endsWith('.sql'))
.sort();
const allFiles = fs.readdirSync(migrationsDir);
const { resolveMigrations } = require('../../migrate');
const files = resolveMigrations(allFiles);

return files.filter((file: string) => !appliedSet.has(file));
}
Expand Down
Loading