Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,9 @@ for all supported AdGuard products.

## Technical Context

- **Language / Version**: TypeScript and JavaScript, Node.js >= 22, ESM (`"type": "module"`)
- **Language / Version**: TypeScript and JavaScript, Node.js >= 22, ESM (`"type": "module"`).
New code must be written in TypeScript. The project is gradually migrating all scripts to
TypeScript; existing `.js` files should be converted to `.ts` when touched.
- **Primary Dependencies**:
- `@adguard/filters-compiler` — compiles filter templates into final filter lists
- `@adguard/agtree` — AdGuard filter rule parser / AST
Expand Down Expand Up @@ -51,7 +53,8 @@ for all supported AdGuard products.
│ └── <sub-target>/ # extension/ only: chromium, chromium-mv3, edge, firefox,
│ # opera, opera-mv3, safari, android-content-blocker, ublock
├── scripts/ # All build and utility scripts
│ ├── build/ # build.js, constants.js, custom_platforms.js, patches.js
│ ├── build/ # build.js, build-config.ts, constants.js,
│ │ # custom_platforms.js, patches.js, strip-generated-meta.ts
│ ├── checksum/ # Checksum generation (index.ts)
│ ├── repository/ # compress.js — repository compression
│ ├── translations/ # Locale download/upload tooling
Expand All @@ -77,8 +80,11 @@ for all supported AdGuard products.
| Command | Description |
| ------- | ----------- |
| `yarn build` | Build all filters (`tsx scripts/build/build.js`) |
| `yarn build:local` | Build filters from cached `filter.txt` files (`tsx scripts/build/build.js --use-cache`) |
| `yarn auto-build` | Full automated build via `bash scripts/auto_build.sh` |
| `yarn build:patches` | Build incremental patches |
| `yarn generate-cache` | Generate cached `filter.txt` from templates (`tsx scripts/build/build.js --generate-cache`) |
| `yarn strip-generated-meta` | Strip generated meta lines from platform filter files |
| `yarn test` | Run unit tests (`vitest run`) |
| `yarn lint` | Run all linters (code + types + markdown) |
| `yarn lint:code` | ESLint check (`eslint . --ext .js,.ts`) |
Expand All @@ -101,6 +107,11 @@ After completing any task that modifies code in `scripts/`:
yarn lint
```

> **Important:** `yarn lint` runs three checks sequentially:
> `yarn lint:code` → `yarn lint:types` → `yarn lint:md`.
> A failure in any one stage aborts the rest. Always verify that **all three**
> pass, including Markdown lint on edited `.md` files.

2. **Run unit tests.** All tests must pass.

```bash
Expand All @@ -124,6 +135,12 @@ After completing any task that modifies code in `scripts/`:
7. **Do not modify third-party filter sources.** Files under `filters/ThirdParty/` are managed
by the upstream filter lists workflow and should not be edited manually.

8. **Synchronise code, tests, and documentation for CLI changes.** When adding or modifying
command-line arguments, build flags, or their compatibility rules:
- Add or update tests in `scripts/build/__tests__/` covering the new behaviour.
- Update the *Command Compatibility* section in `DEVELOPMENT.md`.
- Never merge CLI changes without matching tests and documentation.

## Code Guidelines

### Architecture
Expand Down
125 changes: 125 additions & 0 deletions DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,11 @@
yarn test
```

> **Note:** `yarn lint` runs three checks in sequence: ESLint (`yarn lint:code`),
> TypeScript type checking (`yarn lint:types`), and Markdownlint (`yarn lint:md`).
> All three must pass. The build scripts are a mix of JavaScript and TypeScript;
> `tsx` executes both transparently — no manual compilation step is needed.

## Development Workflow

### Building Filters
Expand Down Expand Up @@ -62,6 +67,123 @@ Generate a build report to a custom file:
yarn build --report='report-adguard.txt'
```

#### Additional Build Flags

The `yarn build` command (and `yarn build:local`) also accepts:

- `--no-patches-prepare` — skip copying `platforms/` to `temp/platforms/`, used
to build patches. Speeds up the build when patch generation
(`yarn build:patches`) is not needed afterwards.
- `--strip-generated-meta` — after compilation, remove generated meta lines
(`! Checksum`, `! Diff-Path`, `! TimeUpdated`, `! Version`) from all filter
files in `platforms/` and `temp/platforms/`. Useful when comparing outputs between builds.

### Generating Filter Cache

To update the cached `filter.txt` files in `filters/`, used for
testing/reproducible builds, run:

```bash
yarn generate-cache
```

This compiles every filter from its `template.txt` and updates the corresponding
`filter.txt` inside `filters/`. Platform-specific filters and patches are **not**
generated. The resulting `filter.txt` files contain the fully resolved filter
content (all `@include` and `!#include` directives expanded) and can be used to
build filters from cache with `yarn build:local`.

### Building From Cache

To build filters from previously cached `filter.txt` files without downloading
external filters, run:

```bash
yarn build:local
```

Under the hood this copies `filters/` to `temp/filters_cached/`, replaces every
`template.txt` with a single `@include "./filter.txt"` directive, and compiles
from that copy. The original `filters/` directory is never modified.

The `-i` / `-s` / `--no-patches-prepare` / `--strip-generated-meta` flags can be
combined:

```bash
yarn build:local -i=1,2,3 --no-patches-prepare --strip-generated-meta
```

**Typical workflow — comparing two compiler versions:**

1. Download and compile filter content into cache:
`yarn generate-cache`
1. Build from cache with generated metadata lines stripped:
`yarn build:local --no-patches-prepare --strip-generated-meta`
1. Rename the output: `mv platforms platforms_A`
1. Switch to the other compiler version
(e.g. `yarn add @adguard/filters-compiler@...`)
1. Build again with the same flags:
`yarn build:local --no-patches-prepare --strip-generated-meta`
1. Rename the output: `mv platforms platforms_B`
1. Diff the two directories (e.g. in Total Commander, WinMerge, or
with `diff -r`)

Both runs use the exact same cached filter content and strip all volatile
metadata, so any difference comes solely from the compiler.

#### Command Compatibility

The following flags can be used with `yarn build` and `yarn build:local`:

- `-i=`, `--include=` — comma-separated filter IDs to build (e.g., `--include=1,2,3`)
- `-s=`, `--skip=` — comma-separated filter IDs to exclude (e.g., `--skip=12,24`)
- `--report=` — custom report file name (e.g., `--report='report-adguard.txt'`)
- `--no-patches-prepare` — skip copying `platforms/` to `temp/platforms/`
- `--strip-generated-meta` — remove volatile metadata lines from built files
- `--use-cache` — build from cached `filter.txt` (same as `yarn build:local`)
- `--generate-cache` — compile filters and update cache only (no platform files)

**Valid combinations:**

```bash
# Base builds
yarn build
yarn build:local

# Filter selection
yarn build --include=1,2,3
yarn build --skip=12,24
yarn build --include=1,2,3 --skip=2 # intersection minus exclusion. Excessive, but it works

# Report output
yarn build --report='report-adguard.txt'

# Patch and metadata control
yarn build --no-patches-prepare
yarn build --strip-generated-meta

# Combined examples
yarn build --include=1,2,3 --no-patches-prepare --strip-generated-meta
yarn build:local --skip=12,24 --report='report.txt' --strip-generated-meta

# Cache generation with filter selection
yarn build --generate-cache
yarn build --generate-cache --include=1,2,3
yarn build --generate-cache --skip=12,24
yarn build --generate-cache --report='report.txt'
```

**Invalid or ineffective combinations:**

```bash
# Mutually exclusive flags → script exits with error
yarn build --use-cache --generate-cache

# --generate-cache exits early; these flags are incompatible → script exits with error
yarn build --generate-cache --strip-generated-meta
yarn build --generate-cache --no-patches-prepare
```

### Automated Build

The `auto-build` script performs a full build with patches and wildcard domain expansion.
Expand Down Expand Up @@ -243,6 +365,9 @@ All build tooling lives under `scripts/`. After making changes:
`scripts/wildcard-domain-processor/__tests__/`.
1. Run `yarn validate` if the change affects filter compilation or platform outputs.

Build scripts under `scripts/` are written in JavaScript and TypeScript, executed
via [tsx](https://github.com/privatenumber/tsx) — no manual compilation step is needed.

## Troubleshooting

### `yarn build` fails with missing platform files
Expand Down
3 changes: 3 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,10 @@
"lint:md": "markdownlint **/*.md",
"test": "vitest run",
"build": "tsx scripts/build/build.js",
"build:local": "tsx scripts/build/build.js --use-cache",
"build:patches": "node scripts/build/patches.js",
"strip-generated-meta": "tsx scripts/build/strip-generated-meta.ts",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yarn strip-generated-meta points to scripts/build/strip-generated-meta.ts, but this file only exports stripGeneratedMetaFromDir() and never invokes it.

I verified locally that the command exits successfully without touching the generated files. Please add a CLI entrypoint here, or point the npm script to a small wrapper that actually calls the function.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7a10cdb

"generate-cache": "tsx scripts/build/build.js --generate-cache",
"validate": "yarn validate:platforms && yarn validate:locales",
"validate:platforms": "tsx scripts/validation/validate_platforms.js",
"validate:locales": "tsx scripts/validation/validate_locales.js",
Expand Down
170 changes: 170 additions & 0 deletions scripts/build/__tests__/build-config.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
import {
describe, it, expect,
} from 'vitest';
// eslint-disable-next-line import/no-unresolved
import { parseFlags, validateFlags, validateArgs } from '../build-config.js';

describe('parseFlags', () => {
it('parses --include and --skip', () => {
const flags = parseFlags(['--include=1,2,3', '--skip=12,24']);
expect(flags.includedFilterIDs).toEqual([1, 2, 3]);
expect(flags.excludedFilterIDs).toEqual([12, 24]);
});

it('parses short forms -i and -s', () => {
const flags = parseFlags(['-i=1,2', '-s=3']);
expect(flags.includedFilterIDs).toEqual([1, 2]);
expect(flags.excludedFilterIDs).toEqual([3]);
});

it('parses --report', () => {
const flags = parseFlags(['--report=report.txt']);
expect(flags.rawReportPath).toBe('report.txt');
});

it('parses all boolean flags', () => {
const flags = parseFlags([
'--use-cache',
'--no-patches-prepare',
'--strip-generated-meta',
]);

expect(flags.useCache).toBe(true);
expect(flags.noPatchesPrepare).toBe(true);
expect(flags.stripGeneratedMeta).toBe(true);
});

it('ignores unknown arguments', () => {
const flags = parseFlags(['--mode=unknown', '--foo']);
expect(flags.useCache).toBe(false);
});

it('filters out NaN from invalid IDs', () => {
const flags = parseFlags(['--include=1,abc,3']);
expect(flags.includedFilterIDs).toEqual([1, 3]);
});
});

describe('validateFlags', () => {
it('allows valid single flags', () => {
const flags = parseFlags(['--use-cache']);
expect(validateFlags(flags)).toBeNull();
});

it('allows --generate-cache with --include', () => {
const flags = parseFlags(['--generate-cache', '--include=1,2,3']);
expect(validateFlags(flags)).toBeNull();
});

it('rejects --use-cache combined with --generate-cache', () => {
const flags = parseFlags(['--use-cache', '--generate-cache']);
const result = validateFlags(flags);
expect(result?.type).toBe('error');
expect(result?.message).toContain('mutually exclusive');
expect(result?.message).toContain('DEVELOPMENT.md');
});

it('errors on --generate-cache combined with --strip-generated-meta', () => {
const flags = parseFlags(['--generate-cache', '--strip-generated-meta']);
const result = validateFlags(flags);
expect(result?.type).toBe('error');
expect(result?.message).toContain('--strip-generated-meta');
expect(result?.message).toContain('incompatible');
expect(result?.message).toContain('DEVELOPMENT.md');
});

it('errors on --generate-cache combined with --no-patches-prepare', () => {
const flags = parseFlags(['--generate-cache', '--no-patches-prepare']);
const result = validateFlags(flags);
expect(result?.type).toBe('error');
expect(result?.message).toContain('--no-patches-prepare');
expect(result?.message).toContain('DEVELOPMENT.md');
});

it('errors on multiple incompatible flags with --generate-cache', () => {
const flags = parseFlags([
'--generate-cache',
'--strip-generated-meta',
'--no-patches-prepare',
]);
const result = validateFlags(flags);
expect(result?.type).toBe('error');
expect(result?.message).toContain('--strip-generated-meta and --no-patches-prepare');
expect(result?.message).toContain('are incompatible');
expect(result?.message).toContain('DEVELOPMENT.md');
});

it('allows multiple valid combinations', () => {
const flags = parseFlags([
'--include=1,2',
'--no-patches-prepare',
'--strip-generated-meta',
]);
expect(validateFlags(flags)).toBeNull();
});
});

describe('validateArgs', () => {
it('allows all valid arguments', () => {
expect(validateArgs(['--include=1,2'])).toBeNull();
expect(validateArgs(['-i=1'])).toBeNull();
expect(validateArgs(['--skip=12'])).toBeNull();
expect(validateArgs(['-s=12'])).toBeNull();
expect(validateArgs(['--report=file.txt'])).toBeNull();
expect(validateArgs(['--use-cache'])).toBeNull();
expect(validateArgs(['--generate-cache'])).toBeNull();
expect(validateArgs(['--no-patches-prepare'])).toBeNull();
expect(validateArgs(['--strip-generated-meta'])).toBeNull();
});

it('rejects random single-word arguments', () => {
expect(validateArgs(['unknown'])).toContain('Unknown argument: unknown');
expect(validateArgs(['build'])).toContain('Unknown argument: build');
expect(validateArgs(['123'])).toContain('Unknown argument: 123');
});

it('includes DEVELOPMENT.md hint in unknown argument error', () => {
const result = validateArgs(['saasdasd']);
expect(result).toContain('DEVELOPMENT.md');
});

it('rejects arbitrary flags', () => {
expect(validateArgs(['--foo'])).toContain('Unknown argument: --foo');
expect(validateArgs(['--arg'])).toContain('Unknown argument: --arg');
expect(validateArgs(['--unknown'])).toContain('Unknown argument: --unknown');
expect(validateArgs(['-a'])).toContain('Unknown argument: -a');
expect(validateArgs(['-x'])).toContain('Unknown argument: -x');
expect(validateArgs(['-1'])).toContain('Unknown argument: -1');
expect(validateArgs(['-Z'])).toContain('Unknown argument: -Z');
});

it('rejects common typos of valid flags', () => {
// Extra letter at end
expect(validateArgs(['--use-cachee'])).toContain('Unknown argument: --use-cachee');
expect(validateArgs(['--generate-cach'])).toContain('Unknown argument: --generate-cach');
expect(validateArgs(['--strip-generate-meta'])).toContain('Unknown argument: --strip-generate-meta');
// Swapped / missing letter
expect(validateArgs(['--usecache'])).toContain('Unknown argument: --usecache');
});

it('rejects near-miss prefixes for value flags', () => {
expect(validateArgs(['--includes=1'])).toContain('Unknown argument: --includes=1');
expect(validateArgs(['--skips=12'])).toContain('Unknown argument: --skips=12');
});

it('rejects --report without =value (bare flag)', () => {
expect(validateArgs(['--report'])).toContain('Unknown argument: --report');
});

it('rejects unknown arguments regardless of position among valid args', () => {
expect(validateArgs(['--include=1', 'junk'])).toContain('Unknown argument: junk');
expect(validateArgs(['junk', '--include=1'])).toContain('Unknown argument: junk');
expect(validateArgs(['--use-cache', '-a', '--skip=2'])).toContain('Unknown argument: -a');
expect(validateArgs(['--genarate-cache']))
.toContain('Unknown argument: --genarate-cache');
});

it('handles empty array (no args)', () => {
expect(validateArgs([])).toBeNull();
});
});
Loading