diff --git a/.markdownlint.jsonc b/.markdownlint.jsonc new file mode 100644 index 00000000..91e9c054 --- /dev/null +++ b/.markdownlint.jsonc @@ -0,0 +1,11 @@ +{ + "MD004": { "style": "dash" }, + "MD010": { "code_blocks": false }, + "MD013": { + "line_length": 80, + "code_block_line_length": 120, + "tables": false, + "headings": false + }, + "MD029": { "style": "ordered" } +} diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 00000000..4e6568dd --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,10 @@ +# Changelog + +## Unreleased + +### Breaking changes + +- Reject legacy DDlog tokens that previously had inconsistent parser treatment: + `typedef`, `bigint`, `bit`, `double`, `float`, `signed`, bare `#`, and `<=>`. + Use `type`, sized integer types such as `u32`/`i64`, `f32`/`f64`, and + `#[...]` attributes instead. diff --git a/docs/contents.md b/docs/contents.md index a664bb40..41a829de 100644 --- a/docs/contents.md +++ b/docs/contents.md @@ -5,6 +5,8 @@ ## Core project documents +- [Changelog](../CHANGELOG.md): User-facing release notes and migration + highlights. - [Repository layout](./repository-layout.md): Map of the repository tree, path responsibilities, and placement rules for contributors. - [Users' guide](./users-guide.md): User-visible syntax support and parser @@ -82,6 +84,8 @@ Plan for transformer declaration grammar alignment. - [2.6.6 resolve relation form coverage](./execplans/2-6-6-resolve-relation-form-coverage.md): Plan for relation form coverage and modelling alignment. +- [2.6.7 finalize legacy token compatibility policy](./execplans/2-6-7-finalize-legacy-token-compatibility-policy.md): + Plan for closing the legacy-token compatibility policy. - [3.1.1 core rule and CST rule traits](./execplans/3-1-1-core-rule-and-cst-rule-traits.md): Plan for lint rule trait foundations. - [3.1.2 rule context struct](./execplans/3-1-2-rule-context-struct.md): diff --git a/docs/ddlint-design.md b/docs/ddlint-design.md index 95f79087..bfa68ca4 100644 --- a/docs/ddlint-design.md +++ b/docs/ddlint-design.md @@ -233,7 +233,10 @@ the syntax-layer contract before crate extraction: the parser now accepts only the canonical spec-form `index Name(field: Type, ...) on Atom`, rejects the older shorthand `index Name on Relation(columns)` with a targeted diagnostic, and exposes the typed field list plus normalized `on` target directly from the -CST-backed `Index` wrapper. +CST-backed `Index` wrapper. Legacy compatibility tokens follow the same +parser-contract rule: `src/parser/reserved_tokens.rs` single-sources +diagnostics for tokens that stay lexed for span precision but have no supported +grammar semantics. Relation declarations use the same CST-backed boundary. The `Relation` wrapper models the declaration preamble with `RelationRole` (`Input`, `Output`, or diff --git a/docs/developers-guide.md b/docs/developers-guide.md index 77bd7505..a78243cc 100644 --- a/docs/developers-guide.md +++ b/docs/developers-guide.md @@ -113,6 +113,11 @@ parsing pipeline. (`D-REL-001` through `D-REL-003`) in `relations/preamble.rs`. - Keep typed consumer-facing relation metadata in `ast/relation.rs`, and inspection-only CST traversal in `ast/relation/inspect.rs`. +- Route reserved-token diagnostics through `src/parser/reserved_tokens.rs`. + That module owns the parser-internal messages and the `rejection_for` + predicate for unsupported legacy tokens. The canonical public policy table + lives in `docs/differential-datalog-parser-syntax-spec-updated.md` section + `9.1`; avoid duplicating it in code comments or local scanner modules. ## Contributor workflow diff --git a/docs/differential-datalog-parser-syntax-spec-updated.md b/docs/differential-datalog-parser-syntax-spec-updated.md index 180191f0..cb16f23a 100644 --- a/docs/differential-datalog-parser-syntax-spec-updated.md +++ b/docs/differential-datalog-parser-syntax-spec-updated.md @@ -90,19 +90,26 @@ ______________________________________________________________________ function calls at parse time. A bare `name(…)` parses as a variable application and is disambiguated later during name resolution. +These case classes describe the intended naming categories. The current parser +uses the generic `T_IDENT` token for identifiers and does not enforce the +initial-case distinction in every context. + ### 2.3 Reserved words and symbols The following **keywords** and **reserved operators** cannot be used as identifiers (final list should be kept 1:1 with the lexer): - **Keywords:** `type`, `function`, `extern`, `transformer`, `input`, `output`, - `relation`,`stream`,`multiset`,`index`,`on`,`primary`,`key`,`apply`,`match`,`if`,`else`,`for`,`in`,`then`,`skip`,`true`,`false`,`var`,`mut`,`return`,`break`,`continue`. + `relation`, `stream`, `multiset`, `index`, `on`, `primary`, `key`, `apply`, + `match`, `if`, `else`, `for`, `in`, `then`, `skip`, `true`, `false`, `var`, + `mut`, `return`, `break`, `continue`, and `as`. - **Special tokens:** `@`, `:-`, `,`, `;`, `:`, `::`, `.`, `&`, `'` (diff marker), `-<` (delay introducer), `=>` (implies), brackets and braces - `()[]{}`. + `()[]{}`, and `#` when it starts an attribute prefix `#[...]`. -Reserved but not part of the grammar: `#`, `<=>`. Implementations must reject -their use with a clear diagnostic. +Reserved but not part of the grammar: `<=>` and bare `#` tokens not followed by +`[` as an attribute prefix. Implementations must reject their use with a clear +diagnostic. #### 2.3.1 Host‑language keyword reservation @@ -184,7 +191,10 @@ intentional. --> **Note:** `++` (concatenation) and `^` (bit‑xor) are part of the operator table and are recognized as operators. `&` in row 13 is expression-only; head -semantics are described in §7.3. +semantics are described in §7.3. `:` is the implemented expression +type-ascription operator, and `as` is the implemented expression cast operator; +both use the type-operator binding level between shifts and bitwise operators. +`as` is also the import alias keyword. ______________________________________________________________________ @@ -209,7 +219,7 @@ Attribute ::= '#[' AttrBody ']' ### 5.2 Imports and types ```ebnf -Import ::= 'import' ScopedPath ';' +Import ::= 'import' ScopedPath ('as' LcName)? ';' Typedef ::= 'type' UcName TypeParams? '=' Type ';' Type ::= UcName TypeArgs? | TupleType | MapType | VecType | Primitive @@ -220,6 +230,10 @@ Primitive ::= 'bool' | 'i8' | 'u8' | 'i16' | … | 'u128' | 'f32' | 'f64' | 'string' | 'interned' ``` +The import alias uses the documented `LcName` category. The current parser +accepts the generic identifier token here and does not enforce that case +restriction. + ### 5.3 Functions and closures ```ebnf @@ -553,19 +567,66 @@ ______________________________________________________________________ ## 9.1 Legacy and compatibility tokens Implementations may encounter historical tokens from older DDlog parsers. This -spec defines their treatment to aid migration: +spec defines their treatment to aid migration. The lexer keeps the token kinds +so diagnostics can point at the exact source span; unsupported uses are +rejected by the parser. - `Aggregate(…)`: accepted and normalized during rule-body semantic extraction to the same canonical `(project, key)` aggregation contract used for `group_by(project, key)`; linters may emit a deprecation diagnostic. -- `FlatMap`/`Inspect`: not language keywords; represent flatmap via RHS pattern - binds instead. If used as keywords, reject with a targeted message. -- `typedef`: not supported; use `type` definitions. Emit an error with a fix - hint. -- Legacy type names such as `bigint`, `bit`, `double`, `float`, `signed`: - not in the grammar. Use sized integer types (`iN`/`uN`), and `f32`/`f64` for - floating‑point. -- `as`: not a keyword in the updated grammar; reject its use as a keyword. +- `FlatMap`/`Inspect`: retained as lexer compatibility tokens; neither is in the + reserved-token rejection set. `FlatMap(…)` is accepted as an identifier-like + RHS call, and both names are accepted in pattern positions. FlatMap-style + binds use RHS patterns. +- `typedef`: rejected. + + ```plaintext + `typedef` is a legacy DDlog keyword; use `type` instead + ``` + +- `as`: accepted as the import alias keyword and the implemented expression + cast operator; `:` is the implemented expression type-ascription operator. +- `bigint`: rejected. + + ```plaintext + `bigint` is a legacy type name; use a sized integer such as `i64` or `u64` + ``` + +- `bit`: rejected. + + ```plaintext + `bit` is a legacy type name; use an unsigned sized integer such as `u32` + ``` + +- `double`: rejected. + + ```plaintext + `double` is a legacy type name; use `f64` + ``` + +- `float`: rejected. + + ```plaintext + `float` is a legacy type name; use `f32` + ``` + +- `signed`: rejected. + + ```plaintext + `signed` is a legacy type name; use a signed sized integer such as `i32` + ``` + +- `#`: accepted only as `#[...]`; bare uses are rejected. + + ```plaintext + `#` is reserved; only `#[...]` attribute syntax is accepted + ``` + +- `<=>`: rejected. + + ```plaintext + `<=>` was reserved upstream but has no semantics in DDlog; remove it + ``` Rationale and resolution status for the aggregation boundary: diff --git a/docs/execplans/2-6-7-finalize-legacy-token-compatibility-policy.md b/docs/execplans/2-6-7-finalize-legacy-token-compatibility-policy.md new file mode 100644 index 00000000..582571cd --- /dev/null +++ b/docs/execplans/2-6-7-finalize-legacy-token-compatibility-policy.md @@ -0,0 +1,929 @@ +# Finalize legacy token compatibility policy + +This ExecPlan (execution plan) is a living document. The sections `Constraints`, +`Tolerances`, `Risks`, `Progress`, `Surprises & Discoveries`, `Decision Log`, +and `Outcomes & Retrospective` must be kept up to date as work proceeds. + +Status: COMPLETE — implementation, validation, and CodeRabbit review are done + +## Purpose / big picture + +Roadmap item `2.6.7` and parser conformance register item `14` are open because +five legacy token classes — `typedef`, `as`, the legacy type names (`bigint`, +`bit`, `double`, `float`, `signed`), `#`, and `<=>` — are recognized by the +tokenizer but receive inconsistent treatment in the parser. Spec section `9.1` +sketches an intent ("reject with an error and fix hint") but stops short of a +closed policy matrix, and spec section `2.3` lists `#` and `<=>` as "reserved +but not part of the grammar" even though `#` is the load-bearing sigil for the +existing attribute syntax `#[...]` and `as` is the load-bearing rename keyword +for `import X as Y`. A novice reading the spec today therefore cannot tell +which uses of these tokens are valid, which are silently accepted, and which +are rejected. + +After this change, a novice should be able to read one short policy table, +predict the parser's behaviour for every legacy token, and either get a +deterministic diagnostic with a fix hint or have the spec confirm the token's +remaining legitimate role. Success is observable when: + +- the parser, unit tests, behavioural tests, and active docs all describe the + same closed policy for every token in scope; +- attempting to use `typedef`, `<=>`, a bare `#` (one not followed by `[`), or + any of the legacy type names in a source program produces a stable, + deterministic diagnostic carrying a token-specific message and a fix hint; +- the existing roles of `as` (in `import` aliases) and `#` (as the attribute + sigil) are explicitly preserved in code, spec, and tests; +- `make check-fmt`, `make lint`, `make markdownlint`, `make nixie`, and + `CI=1 make test` pass after the implementation change; and +- `docs/roadmap.md` item `2.6.7` is marked done only after those checks + succeed. + +## Recommended decision + +Close item `2.6.7` by adopting the "reservation without semantics" pattern +borrowed from Rust's reserved-keyword tier and TypeScript's per-token +diagnostics: keep every token kind in the lexer so spans remain precise, and +reject every use that has no production at the parser stage with a per-token, +deterministic diagnostic including a fix hint. Where a token already has a +load-bearing parser use, the spec is amended to record that use rather than the +parser being weakened to match the older intent. + +The closed policy matrix becomes: + +- `typedef` (`K_TYPEDEF`) — **reject at parse**. Emit + `` `typedef` is a legacy DDlog keyword; use `type` instead ``. The + `src/parser/span_scanners/typedefs.rs` scanner skips the rejected line rather + than producing a `TypeDef` AST node; `collect_reserved_token_errors` in + `src/parser/span_scanner.rs` records the deterministic legacy-keyword error. +- `as` (`K_AS`) — **keep as the import alias keyword and the implemented + expression cast operator**. Spec section `9.1`'s "reject as a keyword" + wording is corrected because upstream DDlog used `as` in both roles and the + current ddlint parser implements both the expression cast and `Import` aliases + (`src/parser/ast/import.rs`, `src/parser/ast/expr.rs`, + `src/parser/span_scanners/imports.rs`). The expression type-ascription + operator remains `:`. Spec section `5.2` is updated so the `Import` + production records the alias clause. +- Legacy type names `bigint`, `bit`, `double`, `float`, `signed` (`K_BIGINT`, + `K_BIT`, `K_DOUBLE`, `K_FLOAT`, `K_SIGNED`) — **reject at parse when used in + type position**. Each token gets a fix hint naming the modern sized type + (`bigint → use a sized integer such as i64 or u64`, + `bit → use unsigned sized types such as u32`, `double → use f64`, + `float → use f32`, + `signed → use the signed sized integer types such as i32`). Tokens + encountered outwith a type position still produce one deterministic message + that names the token, so an identifier such as `let bigint = ...` cannot + silently shadow the keyword. +- `#` (`T_HASH`) — **keep as the attribute sigil; reject bare uses**. A + `T_HASH` followed by `T_LBRACKET` enters the existing attribute scanner in + `src/parser/span_scanners/attributes.rs` unchanged. Every other occurrence is + rejected with + `` `#` is reserved; only `#[...]` attribute syntax is accepted``. Spec section + `2.3` is updated so `#` is listed as a special attribute-prefix token and + the "reserved but not part of the grammar" line is narrowed to bare uses. +- `<=>` (`T_SPACESHIP`) — **reject at parse with a fix hint**. Emit + `` `<=>` was reserved upstream but has no semantics in DDlog; remove it``. + The Pratt expression layer in `src/parser/expression/` is the natural site to + detect and recover. + +All diagnostics flow through a new module `src/parser/reserved_tokens.rs`. The +module hosts the per-token message constants, a single classification predicate +`rejection_for(kind: SyntaxKind) -> Option<&'static str>` that returns the +message text when the kind is a rejected reserved token (and `None` otherwise), +and a thin constructor that builds a `Simple::custom` from a span +plus a message. Every enforcement path routes through this predicate so the +eight diagnostic messages stay aligned; the bare-`#` message is selected by +contextual lookahead. The module name uses "reserved tokens" rather than +"legacy tokens" because `as` is also a legacy token but is *kept*; the rejected +set is more precisely described as "reserved without semantics" in the +Rust-tier sense. + +The parser emits these as `Simple::custom` failures so they appear +in `Parsed::errors` like every other deterministic parser diagnostic and +benefit from the existing `into_owned`/clone machinery. The message constants +are `pub(crate)` and the messages themselves are *not* a public contract: their +wording may change in any release. Tests import the constants by name so test +breakage flags any drift. If a future milestone exposes structured diagnostic +codes for IDE tooling, the codes — not the messages — become the public +contract; this plan does not introduce such codes. + +This is the smallest decision that simultaneously (1) closes the conformance +delta in code, (2) reconciles the contradictions inside the spec, and (3) +avoids breaking the already-shipping `import X as Y` and `#[attr]` surfaces. + +If implementation reveals a workspace consumer that depends on parsing +`typedef`, `bigint`, `bit`, `double`, `float`, `signed`, bare `#`, or `<=>` as +valid grammar (e.g., a parser test asserting that a `typedef` declaration +produces a `TypeDef` AST node, or a fixture using `bigint` as a type name), +update that consumer in the same change. The plan budgets for this in the +`typedef` migration step. If any such consumer cannot be updated within the +scope tolerance below, stop and escalate. + +## Constraints + +- Align the active written contract in + `docs/differential-datalog-parser-syntax-spec-updated.md` sections `2.3`, + `5.2`, and `9.1` to the final parser behaviour. +- Keep the scope centred on the five token classes named above. Do not fold + relation form, brace-group, or wider terminator policy work from roadmap items + `2.6.6` and `2.6.8` into this change. +- Preserve the existing AST surface for valid programs. Specifically, the + `Import` typed wrapper keeps its `alias` accessor and the attribute parser + keeps consuming `T_HASH` `T_LBRACKET` sequences without behaviour change. +- Replace the `typedef`-accepting code path with a deterministic rejection + path; do not leave a hidden second entrypoint that still produces a `TypeDef` + AST node from `typedef ...`. +- Token kinds in `src/tokenizer.rs` remain unchanged; this milestone is a + parser policy decision, not a lexer redesign. +- Add both unit tests in `src/parser/tests/` and behavioural tests in + `tests/` covering each token's reject path, plus regression coverage that + proves `as` aliases and `#[attr]` placements still parse. +- Update the relevant design documents, not only the conformance register: + `docs/differential-datalog-parser-syntax-spec-updated.md`, + `docs/parser-conformance-register.md`, `docs/parser-implementation-notes.md`, + `docs/ddlint-design.md`, `docs/users-guide.md` (because programs that + previously parsed now produce errors), `docs/developers-guide.md` (because + there is a new internal reserved-token diagnostic convention), and + `CHANGELOG.md` (or the closest equivalent migration-notes file, so adopters + scanning a release note find the breaking changes without having to browse + the users' guide). The canonical home for the per-token policy table is spec + section `9.1`; the developers' guide references that section rather than + duplicating the table. +- Mark `docs/roadmap.md` item `2.6.7` done only after all documentation, + implementation, and validation steps succeed. +- Run repository quality gates through Make targets using `set -o pipefail` + and `tee` so truncated logs cannot hide regressions. +- Keep comments and documentation in en-GB-oxendict spelling. + +## Tolerances (exception triggers) + +- Scope: the original 14-file tolerance was exceeded during the + pre-implementation audit and the human operator approved proceeding on + 2026-06-26. Continue only for files directly required by this plan. Stop and + escalate if the implementation exceeds 28 edited files, 800 net new code + lines, or pulls in relation-form, brace-group, or unrelated terminator policy + work. +- Interface: if the only coherent fix requires removing + `Import::alias()`, reshaping the attribute scanner, or changing the public + `Parsed` surface, stop and escalate. +- Compatibility: if non-test workspace code depends on parsing `typedef`, + bare `#`, `<=>`, or any of the legacy type names as valid syntax, stop and + document those callers before changing behaviour. +- Grammar: if updating spec sections `2.3`, `5.2`, or `9.1` leaves an + unavoidable contradiction with sections `4` (operator table) or `5.5`–`5.8`, + stop and escalate with concrete section references instead of silently + widening the task. +- Diagnostics: every reject path must route through + `reserved_tokens::rejection_for` and the shared constructor. If any + enforcement site emits a `Simple::custom` for a rejected reserved + token without going through the predicate, stop and refactor before adding + more sites. +- Visibility: the per-token message constants stay `pub(crate)`. If any + consumer outwith the crate needs to match against the strings, stop and + introduce structured diagnostic codes rather than widening visibility. +- Validation: if `make check-fmt`, `make lint`, or `make test` still fails + after three focused fix rounds, stop and escalate with the specific failing + targets. +- Runner stability: if `make test` hits the known repository-wide nextest + stall rather than a deterministic regression from this task, capture the + evidence and escalate instead of silently substituting a different profile. + +## Risks + +- Risk: removing `typedef` acceptance breaks existing parser tests that + positively assert a `TypeDef` AST node from `typedef ...` input, and may + break sample programs under `tests/` or `examples/`. Severity: high. + Likelihood: high. Mitigation: audit `src/parser/tests/parser.rs` lines + 272–293 and run `rg "typedef\b" -t md -t rs` before changing the scanner; + migrate any `typedef`-shaped fixtures to `type` form in the same commit that + introduces the rejection. +- Risk: legacy type names such as `float` are common identifiers in + user-authored fixtures and could cause unrelated test failures when the + rejection lands. Severity: medium. Likelihood: medium. Mitigation: grep for + each legacy type name across the workspace before tightening the parser, and + update the spec's "reserved words" example list in section `2.3` so + contributors understand the constraint. +- Risk: the spec is internally inconsistent: section `2.3` says `#` is + reserved-but-not-grammar while section `9.1` says only legacy tokens outside + the grammar are rejected, and the attribute grammar in section `5.1` already + uses `#[...]`. Severity: high. Likelihood: high. Mitigation: resolve the + contradiction in one atomic spec change in stage C so reviewers see the + corrected grammar as a single coherent contract. +- Risk: emitting one `Simple::custom` per token without a shared + helper risks message drift between scanners and tests. Severity: medium. + Likelihood: medium. Mitigation: centralize the eight messages and their fix + hints, including the contextual bare-`#` message, in a single module + (`src/parser/reserved_tokens.rs`) and reference the same constants from + scanners and tests. +- Risk: the `<=>` operator can appear in user code at lex points where the + Pratt parser is not active (e.g., inside a relation row when expression + parsing has bailed out). Severity: medium. Likelihood: low. Mitigation: add a + span-level fallback in the top-level span scanner so a stray `T_SPACESHIP` + outwith expression context still emits the deterministic diagnostic. +- Risk: changing parser behaviour silently breaks downstream linter rules + that assumed a `TypeDef` AST node would be present for legacy fixtures. + Severity: medium. Likelihood: low. Mitigation: search `src/linter/` and + `src/sema/` for any reliance on `typedef`-shaped fixtures before merging, and + add one explicit regression test under `src/linter/tests/` (or the closest + existing home) that feeds a `typedef`-only programme through the linter and + asserts the rejection diagnostic appears in `Parsed::errors` *and* that no + linter rule produces a false-negative pass on the file. + +## Progress + +- [x] (2026-05-29) Reviewed roadmap item `2.6.7`, parser conformance register + item `14`, the syntax spec section `9.1`, parser-implementation-notes, the + tokenizer, every parser site that mentions one of the five token kinds, and + the existing diagnostic catalogue (`src/parser/error_messages.rs`). +- [x] (2026-05-29) Researched upstream DDlog parser treatment in + `vmware/differential-datalog` and reserved-keyword diagnostic patterns in + rustc, TypeScript, and Swift to ground the recommended decision. +- [x] (2026-05-29) Drafted this ExecPlan in + `docs/execplans/2-6-7-finalize-legacy-token-compatibility-policy.md`. +- [x] (2026-05-29) Ran the Logisphere community-of-experts review on the + draft, recorded a verdict of "proceed with conditions", and revised the plan: + renamed the policy module to `reserved_tokens`, pinned the message constants + to `pub(crate)`, added a single classification predicate routed through every + enforcement site, recorded the workspace audit counts, committed to a linter + / sema regression test, added a `CHANGELOG.md` update, and recorded the + rejected lex-stage-rejection alternative in the Decision Log. +- [x] (2026-06-26) Add red tests in `src/parser/tests/reserved_tokens.rs` for + each reject path and regression tests for the preserved `as` and `#[...]` + uses. +- [x] (2026-06-26) Implement the reserved-token diagnostic module and route + full-program diagnostics through `collect_reserved_token_errors` in the + top-level span scanner, with expression parsing using the same predicate and + constructor. +- [x] (2026-06-26) Update active documentation: spec sections `2.3`, `5.2`, and + `9.1`; the conformance register; parser-implementation-notes; ddlint-design; + users' guide; developers' guide. +- [x] (2026-06-26) Run `make fmt`, `make markdownlint`, `make nixie`, + `make check-fmt`, + `make lint`, and `CI=1 make test`, capturing each log under `/tmp`. +- [x] (2026-06-26) Mark roadmap item `2.6.7` done and close the conformance + register entry as `implemented`. +- [x] (2026-06-16) Resumed implementation on branch + `2-6-7-finalize-legacy-token-compatibility-policy`, loaded `leta`, + `execplans`, `rust-router`, `rust-unit-testing`, and `rust-errors`, and + re-read the plan and active documentation index before editing code. +- [x] (2026-06-16) Re-ran the workspace token audit before stage A and found + that completing the plan as written will require more than the current + 28-file scope tolerance because documentation updates, parser/test changes, + and owned example migrations together exceed that threshold. +- [x] (2026-06-26) Resumed implementation after explicit human direction to + proceed past the prior blocked state, loaded `leta`, `execplans`, + `rust-router`, `rust-errors`, `rust-unit-testing`, `nextest`, and + `commit-message`, confirmed the branch tracks + `origin/2-6-7-finalize-legacy-token-compatibility-policy`, updated the PR + title to remove the `Plan:` prefix, renamed the Lody session, and recorded + the raised scope tolerance. +- [x] (2026-06-26) Ran the initial documentation gates for the resumed execplan + update. `make fmt` established a one-time Markdown formatting baseline across + existing documents; `make markdownlint` and `make nixie` then passed. +- [x] (2026-06-26) Observed the red stage with `cargo test reserved_tokens`: ten + reserved-token policy tests failed for the expected missing diagnostics or + missing `type` support. +- [x] (2026-06-26) Added `src/parser/reserved_tokens.rs`, rejected unsupported + tokens through the top-level parser and Pratt expression path, switched + regular type-definition support from `typedef` to `type`, migrated owned + examples and fixtures from `typedef` / `bit`, and added + `tests/reserved_token_rejection.rs`. +- [x] (2026-06-26) Verified the focused suites: + `cargo test reserved_tokens`, `cargo test --lib parser::tests`, and + `cargo test --test reserved_token_rejection` all passed. +- [x] (2026-06-26) Fixed stale `typedef` fixtures in attribute-placement and + name-uniqueness tests after the first full `CI=1 make test` run exposed + deterministic failures. The attribute scanner now treats `type` as the valid + target and `typedef` as a rejected legacy keyword. +- [x] (2026-06-26) Final implementation gates passed with logs captured under + `/tmp/2-6-7-final-make-{fmt,markdownlint,nixie,check-fmt,lint,test}.log`. +- [x] (2026-06-26) Ran `coderabbit review --agent` after pushing commit + `405e5e1`; CodeRabbit completed with `findings: 0`. + +## Surprises & Discoveries + +- The current parser already *accepts* `typedef` and exposes the result via + `root.type_defs()` in `src/parser/tests/parser.rs` (lines 272–282). Closing + this conformance item therefore requires changing existing positive test + assertions, not just adding new reject tests. +- The spec is internally inconsistent on `#`: section `2.3` declares it + reserved-but-not-grammar while the attribute grammar (section `5.1`) and the + parser both depend on `#[...]`. The same contradiction does not appear for + `<=>`, which has no live use anywhere in the parser. +- Upstream DDlog treats `as` as a real keyword in both `import X as Y` and + the `expr as type` cast. Section `9.1`'s "reject as a keyword" wording + therefore conflicts with both upstream prior art and the live ddlint parser; + the only safe reading is that section `9.1` was written before the `Import` + alias clause was preserved. +- The diagnostic catalogue (`src/parser/error_messages.rs`) is currently + two constants long and was added only for transformer-specific messages. It + sits alongside the new `src/parser/reserved_tokens.rs`; both are + parser-internal diagnostic homes and the developers' guide notes the + distinction (per-feature messages vs the reserved-token policy module). +- Workspace audit counts (pre-implementation): `\btypedef\b` appears 9 + times across 7 `examples/*.dl` fixture files and 0 times in `*.rs` source + programmes (the `.rs` matches are tokenizer-table entries and code + identifiers, not parsed input). The legacy type-name family + `\b(typedef|bigint|bit|double|float|signed)\b` totals 22 matches across 9 + `examples/*.dl` files; sampling those files is required during stage A to + distinguish identifier-position uses (none expected after the + upstream-aligned grammar) from type-position uses that will need migration. + `<=>` does not appear in any `.rs` source programme outwith the tokenizer + itself, and a manual scan of `examples/*.dl` is still required for + completeness. These counts confirm the approved scope tolerance (≤28 files, + ≤800 net LOC) is realistic provided the `.dl` examples are updated alongside + the parser change. +- On 2026-06-16, `docs/repository-layout.md`, referenced by the general + repository-orientation guidance, was absent. Orientation used + `docs/contents.md` and `leta files` instead. +- On 2026-06-26, `docs/users-guide.md`, referenced by the documentation update + requirement, was also absent. The implementation must either create the + missing user-facing guide deliberately or record why the existing migration + notes are the closest active user-facing home for this change. +- On 2026-06-16, a refined audit of parseable `.dl` examples found legacy + tokens in nine owned examples: `examples/left_join_by_negation.dl`, + `examples/tuple_destructuring.dl`, `examples/paths_excluding.dl`, + `examples/reachability.dl`, `examples/functions_and_match.dl`, + `examples/extern_transformer_decl.dl`, `examples/hello_join.dl`, + `examples/ref_and_intern.dl`, and `examples/primary_key_and_index.dl`. + `typedef` appears in seven examples; `bit` appears in three examples. No + `<=>`, `bigint`, `double`, `float`, `signed`, or bare `#` use was found in + the example programmes. +- No `CHANGELOG.md` exists in the repository. The closest existing + migration-notes file found by `rg --files` is + `docs/differential-datalog-parser-syntax-spec-migration-plan.md`, but using + it as release-facing migration notes would be a new documentation role and + should be a conscious decision. +- On 2026-06-26, the required `make fmt` gate normalized Markdown wrapping and + table spacing in existing documents beyond this plan. This formatter baseline + is mechanical and is recorded separately from parser-policy scope. +- The implementation revealed that the existing scanner accepted `typedef` but + did not collect modern `type` aliases. The green implementation therefore + both rejects `typedef` and makes `type Foo = ...` produce the existing + `TypeDef` AST wrapper. +- The `make markdownlint` target invokes `markdownlint`, while the repository + only had `.markdownlint-cli2.jsonc`. Adding `.markdownlint.jsonc` with the + same rule settings made the target honour the documented "do not wrap tables" + rule instead of reporting table rows from the formatter baseline as `MD013` + paragraph violations. +- Full-suite validation found two stale `typedef` assumptions outside the new + reserved-token tests: attribute placement still treated `typedef` as a + permitted target, and name-uniqueness duplicate fixtures expected a second + `typedef` declaration to build a `TypeDef` node. Both were migrated to modern + `type` where the test was not itself about legacy-token rejection. + +## Decision Log + +- Decision: adopt Rust's "reserved keyword" pattern — lex into named token + kinds, reject at the parser stage with a per-token diagnostic plus fix hint — + rather than dropping the token kinds from the lexer. Rationale: preserves + precise spans, keeps editor highlighters working, mirrors upstream DDlog's + `reservedNames`/`reservedOpNames` model, and matches rustc, TypeScript, and + Swift's stage choice. +- Decision: keep `as` as a live keyword for the `Import` alias clause and + implemented expression cast operator, and update spec section `9.1` to record + this. The expression type-ascription operator remains `:`. Rationale: + upstream DDlog used `as` in both roles, the current ddlint parser already + depends on it for `Import`, and removing it would break any programme that + uses `import foo::bar as baz`. +- Decision: keep `#` as the attribute sigil and reject only bare uses (a + `T_HASH` not immediately followed by `T_LBRACKET`). Rationale: aligns with + upstream DDlog's `#[...]` attribute syntax, preserves + `src/parser/span_scanners/attributes.rs` unchanged for its happy path, and + resolves the section `2.3` vs section `5.1` spec contradiction by narrowing + rather than widening the rejection rule. +- Decision: replace `typedef` acceptance with deterministic rejection and + migrate any positive fixtures in the same change. Rationale: the spec + explicitly directs users to `type`, and continuing to silently accept + `typedef` while documenting it as a legacy keyword would leave the + conformance register stuck at `scheduled` indefinitely. +- Decision: centralize the eight diagnostic messages and fix hints, with the + bare-`#` message selected contextually, in a single Rust module rather than + copy-pasting them across scanners. Rationale: keeps wording aligned with + tests, prevents drift, and matches the pattern already established by + `error_messages.rs` for the transformer messages. +- Decision: this milestone does not touch the wider top-level statement + terminator policy, the relation-form work in `2.6.6`, or the brace-group + decision in `2.6.8`. Rationale: those items are tracked separately and + folding them in here would push past the scope tolerance. +- Decision: rejected — lex-stage rejection (mapping the rejected tokens + directly to `N_ERROR` in the tokenizer). Rationale: the parser would lose the + named kind it needs to produce a structural fix hint (e.g., "did you mean + `type Foo = u32`?"), recovery would degrade because every error becomes the + same `N_ERROR` shape, and editor highlighters would lose their per-keyword + colouring for legacy tokens. Parse-stage rejection preserves the kind, + mirrors rustc, TypeScript, and Swift, and adds only a single classification + predicate to the parser. The simpler lex-stage approach was considered and is + intentionally not adopted. +- Decision: name the module `reserved_tokens`, not `legacy_tokens`. + Rationale: `as` is also a legacy DDlog token but is *kept*, so placing it + under a "legacy" banner would mislead future contributors. The rejected set + is precisely the "reserved without semantics" tier from the Rust + reserved-keyword model, so the module name reflects that pattern. +- Decision: cut `typedef` over immediately rather than running a + one-release deprecation warning. Rationale: the workspace audit shows + `typedef` use is confined to `examples/*.dl` fixtures owned by this + repository, so a single coordinated migration is cheaper than carrying a + deprecation cycle and the conformance register entry stuck at `scheduled`. If + subsequent adopter feedback shows external programmes depend on `typedef`, + the policy module can grow a warning variant without changing the public + surface. +- Decision point recorded before implementation: the plan's original 14-file + scope tolerance was exceeded before implementation began. Completing the plan + exactly as written appeared to require at least parser code, new parser + tests, a behavioural test, a linter regression, seven active documentation or + migration-note files, the roadmap, this ExecPlan, and up to nine owned + example migrations. Options were: + 1. raise the file-count tolerance and proceed with the complete plan; + 2. keep the tolerance and narrow the milestone by deferring owned example + migration or some documentation updates, accepting that the repository + may temporarily contain invalid examples or incomplete public contract + text; + 3. split the work into multiple explicitly scoped commits or follow-up + ExecPlans while keeping this branch blocked at the parser-policy + boundary. + The cleanest option is to raise the tolerance because the extra files are not + scope creep; they are direct consequences of the already-approved acceptance + criteria. +- Decision: proceed with the complete plan after human approval on + 2026-06-26 and raise the scope tolerance to 28 edited files / 800 net new + code lines while keeping the grammar scope unchanged. Rationale: the + additional files are direct consequences of the approved acceptance criteria + (tests, owned examples, and active documentation), not adjacent parser work. +- Decision: exclude the one-time `make fmt` Markdown baseline from the + parser-policy file-count tolerance while keeping all semantic parser, + example, and documentation edits inside the raised scope. Rationale: the + formatter baseline is required by the repository gate and does not widen the + grammar or diagnostic policy under implementation. +- Decision: create `docs/users-guide.md` and `CHANGELOG.md` instead of treating + parser migration notes as user-facing release notes. Rationale: both files + were absent, and creating explicit homes keeps the legacy-token migration + visible without changing the purpose of older parser migration documents. +- Decision: add `.markdownlint.jsonc` mirroring the existing + `.markdownlint-cli2.jsonc` rule configuration. Rationale: the Makefile uses + the `markdownlint` CLI directly, and this config file lets the required gate + enforce the repository's documented Markdown rules consistently. + +## Outcomes & Retrospective + +Implementation now matches the closed policy matrix in spec section `9.1`. +`typedef`, legacy type names, bare `#`, and `<=>` produce deterministic parser +diagnostics through `collect_reserved_token_errors` in +`src/parser/span_scanner.rs`, using `src/parser/reserved_tokens.rs`; `type`, +`import X as Y`, and `#[...]` remain accepted. The owned examples and +parser/linter fixtures were migrated away from legacy `typedef` and `bit` +syntax. + +The final deterministic gates passed on 2026-06-26: + +- `make fmt` +- `make markdownlint` +- `make nixie` +- `make check-fmt` +- `make lint` +- `CI=1 make test` + +CodeRabbit reviewed the pushed implementation commit and reported zero findings. + +The main implementation lesson was that `typedef` compatibility had leaked into +tests whose purpose was not compatibility: attribute placement and +name-uniqueness both needed fixture updates so they exercise the modern grammar +rather than depending on a legacy alias. + +## Context and orientation + +The relevant code and documents are concentrated in a small set of files. A +novice with only this ExecPlan and the current working tree should be able to +locate each one without prior context. + +- `src/tokenizer.rs` — defines `K_TYPEDEF`, `K_AS`, `K_BIGINT`, `K_BIT`, + `K_DOUBLE`, `K_FLOAT`, `K_SIGNED`, `T_HASH`, and `T_SPACESHIP`. The `KEYWORDS` + `phf` map (lines around `120`–`195`) and the `Token` enum (lines around + `60`–`112`) are the source of truth for the token names used elsewhere. +- `src/parser/span_scanners/typedefs.rs` — handles the rejected `typedef` line + by skipping it, without recording a `TypeDef` span. The deterministic + diagnostic is added by `collect_reserved_token_errors` in + `src/parser/span_scanner.rs`. +- `src/parser/span_scanners/imports.rs` and `src/parser/ast/import.rs` — + consume `K_AS` to record the optional alias on `Import` AST nodes. Both files + must stay green; `as` is *not* being rejected. +- `src/parser/span_scanners/attributes.rs` — consumes `T_HASH` `T_LBRACKET`. + The happy path stays unchanged; the new bare-`#` rejection lives in the + top-level span scanner so it sees uses outwith attribute context. +- `src/parser/expression/` — the Pratt parser and infix table. A stray + `T_SPACESHIP` reaches this layer when expression parsing is active and is the + natural site to detect and recover. +- `src/parser/error_messages.rs` — current per-feature diagnostic + catalogue (two transformer-specific constants today). This module is left in + place and not extended; reserved-token messages live in a sibling module so + per-feature and policy-level diagnostics stay visually distinct. +- `src/parser/reserved_tokens.rs` — new module added by this milestone. + Hosts the eight `pub(crate)` message constants, including the contextual + bare-`#` message, the `rejection_for(kind) -> Option<&'static str>` + predicate, and the `Simple::custom` constructor that every + enforcement site shares. +- `src/parser/tests/parser.rs` (lines 272–293) — currently asserts that + `typedef` declarations parse into `TypeDef` nodes; these assertions are + inverted in stage A. +- `src/parser/tests/` — feature-specific unit tests. A new file + `src/parser/tests/reserved_tokens.rs` is the natural home for the eight + reserved-token rejection cases, with bare-`#` selected by context. +- `tests/` — `tests/attribute_placement.rs` and `tests/name_uniqueness.rs` + show the canonical shape for behavioural unhappy-path coverage. A new + `tests/reserved_token_rejection.rs` files the end-to-end coverage. +- `docs/differential-datalog-parser-syntax-spec-updated.md` — spec sections + `2.3`, `5.2`, and `9.1` are updated in stage C. The grammar table in section + `5.2` gains the alias clause; section `2.3` narrows the bare-`#` rejection; + section `9.1` records the final per-token policy table. +- `docs/parser-conformance-register.md` — item `14` is rewritten and marked + `implemented` once code, tests, and spec agree. +- `docs/parser-implementation-notes.md` — gains a short parser-boundary + note describing the reserved-token diagnostic module and the fact that + rejection happens at parse rather than lex. +- `docs/ddlint-design.md` — gains one sentence in the parser-contract + section recording the per-token reservation policy by name. +- `docs/users-guide.md` — gains a short note describing the new errors a + user may see when upgrading legacy DDlog input. +- `docs/developers-guide.md` — gains a short note describing the internal + reserved-token diagnostic convention so future contributors emit messages + through the same module. + +At the start of this work, the live contradictions are: + +- Spec: "`typedef`: not supported; emit an error with a fix hint" (section + `9.1`). +- Code: `typedef` is accepted and produces a `TypeDef` AST node + (`src/parser/span_scanners/typedefs.rs`, `src/parser/tests/parser.rs`). +- Spec: "`as`: not a keyword in the updated grammar" (section `9.1`). +- Code: `as` is the import alias keyword + (`src/parser/span_scanners/imports.rs`, `src/parser/ast/import.rs`). +- Spec: "Reserved but not part of the grammar: `#`, `<=>`" (section `2.3`). +- Code: `#` is the attribute sigil (`src/parser/span_scanners/attributes.rs`) + and `<=>` is tokenized but never consumed. +- Spec: legacy type names "not in the grammar" (section `9.1`). +- Code: legacy type names are tokenized into keyword kinds but never + consumed; no diagnostic is emitted today. + +The implementation resolves each of these mismatches atomically in code, tests, +and docs. + +## Plan of work + +### Stage A: audit the workspace and establish red tests for the chosen policy + +Begin with a workspace audit so the migration scope is known before any parser +change lands. Run `rg` across `*.rs`, `*.md`, `*.dl`, and `*.ddlog` for +`\btypedef\b`, each legacy type name, bare `#`, and `<=>`. Append the per-file +results to the Surprises section so reviewers see the migration list. The +pre-implementation counts already recorded there are the starting baseline. + +Then change tests so they describe the chosen grammar before touching scanner +logic. + +- Add `src/parser/tests/reserved_tokens.rs` containing rstest cases for + every reject path: + - `typedef Foo = u32;` produces a deterministic legacy-keyword diagnostic + and no `TypeDef` AST node. + - `1 <=> 2` inside an expression context produces the deterministic + spaceship-rejection diagnostic. + - `# foo` (bare hash, not `#[...]`) produces the deterministic bare-hash + diagnostic. + - Each legacy type name appearing in a type position (e.g., + `type Foo = bigint;`) produces the matching deterministic diagnostic + with the modern-type fix hint. +- Invert the existing positive assertions in + `src/parser/tests/parser.rs` lines 272–293 so `typedef` inputs are expected + to fail with the new diagnostic message rather than succeed. +- Add a regression case in the same file asserting that + `import foo::bar as baz` still parses cleanly and that `Import::alias()` + returns `Some("baz")`. +- Add a regression case asserting that `#[attribute] type Foo = u32;` still + parses cleanly and produces both an attribute and a `TypeDef`. +- Add a parser-level integration assertion in `src/parser/tests/parser.rs` + and the shared fixtures (`src/parser/tests/programs.rs`, + `src/parser/tests/specs.rs`) so the broader parser suite encodes the same + contract. + +At the end of stage A the new unit suite should fail for the right reason +(rejection diagnostics not yet emitted) and the regression cases should still +pass (because `as` and `#[...]` paths have not changed yet). + +### Stage B: implement deterministic grammar handling + +Adjust the parser so the chosen contract is enforced explicitly and recoverably. + +- Add `src/parser/reserved_tokens.rs` with eight `pub(crate)` message + constants, the `rejection_for(kind: SyntaxKind) -> Option<&'static str>` + predicate, and a `reserved_token_error(span, message) -> Simple` + constructor. The module doc string explains the "reservation without + semantics" pattern and links to spec section `9.1`. +- Keep `handle_typedef` in `src/parser/span_scanners/typedefs.rs` on the + line-skipping recovery path rather than producing a `TypeDef` span. The + top-level `collect_reserved_token_errors` call in + `src/parser/span_scanner.rs` classifies `K_TYPEDEF` through + `reserved_tokens::rejection_for` and builds the diagnostic through the shared + constructor. +- Extend the top-level span scanner (`src/parser/span_scanner.rs` or the + appropriate dispatcher in `src/parser/span_scanners/`) so every token drawn + from the stream is first classified via `reserved_tokens::rejection_for`. If + it returns `Some(message)`, the scanner records the diagnostic and skips the + offending line. Bare `T_HASH` (a `T_HASH` not immediately followed by + `T_LBRACKET`) is classified explicitly because the attribute-prefix case must + still reach the attribute scanner unchanged. +- Extend the Pratt expression layer in `src/parser/expression/infix.rs` + (or `pratt.rs`) so `T_SPACESHIP` and any legacy type-name keyword encountered + inside an expression context routes through the same predicate and recovers + to the next infix boundary. +- Extend the type-position scanners (the bodies of `TypeDef`, parameter + lists, and field declarations) so the five legacy type names route through + the same predicate, producing the matching diagnostic with the modern-type + fix hint. No site constructs its own `Simple::custom` for a + rejected reserved token; every site goes through + `reserved_tokens::reserved_token_error`. +- Keep recovery behaviour aligned with existing scanners so subsequent + declarations still parse cleanly. + +### Stage C: align documentation and public contract text + +Once parser and tests agree, update every active document that carries the +grammar contract. + +- In `docs/differential-datalog-parser-syntax-spec-updated.md`: + - Section `2.3`: list `#` as a special attribute-prefix token, retain + `<=>` in the reserved list, and narrow the "reject" sentence so it + covers `<=>` and bare `#` only. + - Section `5.2`: record the optional `('as' LcName)?` alias clause. `LcName` + is the documented grammar category; the parser's generic identifier + parser does not enforce the case distinction. + - Section `9.1`: replace the loose prose with a closed policy table + enumerating each token, the reject site, the diagnostic message + template, and the fix hint. +- In `docs/parser-conformance-register.md` item `14`: rewrite the + current/spec/target paragraphs so they match the final policy, then mark the + entry `implemented`. +- In `docs/parser-implementation-notes.md`: add a short parser-boundary + note describing the reserved-token diagnostic module and explaining why + rejection happens at parse rather than lex. +- In `docs/ddlint-design.md`: add one sentence in the parser-contract + section recording the per-token reservation policy and naming the + reserved-token diagnostic module. +- In `docs/users-guide.md`: add a short subsection ("Legacy DDlog tokens") + describing what programmes encounter when porting from upstream DDlog, with + one line per token and its replacement. +- In `docs/developers-guide.md`: add a short subsection + ("Reserved-token diagnostics") describing the convention so future + contributors route messages through `src/parser/reserved_tokens.rs`, and + linking to spec section `9.1` as the canonical policy table rather than + duplicating it. +- In `CHANGELOG.md` (or the closest migration-notes file): add a + "Breaking changes" entry naming each newly rejected token, the replacement, + and a one-line example of the diagnostic users will see. + +### Stage D: add behavioural coverage for end-to-end parser behaviour + +Add behavioural tests in `tests/` so the public `parse()` entrypoint proves the +decision, not just the internal unit suites. + +- Add `tests/reserved_token_rejection.rs` with one rstest case per token + exercising a complete programme through `parse()` and asserting the presence + of the matching diagnostic in `Parsed::errors`. +- Include a behavioural test in the same file that asserts + `import foo::bar as baz` still produces an `Import` AST node with the + expected alias. +- Include a behavioural test asserting that `#[attribute] type Foo = u32;` + still produces both an attribute and a `TypeDef` and that the absence of + attribute brackets after `#` produces the bare-hash diagnostic. +- Add one explicit linter / sema regression test (under + `src/linter/tests/` or `src/sema/tests/`, whichever already hosts similar + coverage) that feeds a `typedef Foo = u32;`-only programme through the linter + pipeline and asserts (a) the rejection diagnostic appears in `Parsed::errors` + and (b) no existing linter rule produces a false-negative clean pass on the + file. This guards against the Doggylump pre-mortem scenario where a rule + silently misses now-error input because the `TypeDef` AST node it depended on + has disappeared. + +### Stage E: validation and close-out + +After implementation and documentation updates: + +1. Run `make fmt`. +2. Run `make check-fmt`. +3. Run `make lint`. +4. Run `CI=1 make test`. +5. Run `make markdownlint`. +6. Run `make nixie`. +7. Only after those gates pass, update `docs/roadmap.md` to mark item + `2.6.7` done. +8. Update this ExecPlan's `Progress`, `Decision Log`, and + `Outcomes & Retrospective` sections. +9. Run `coderabbit review --agent` and clear every concern before closing + the PR for review. + +## Concrete steps + +1. Audit the workspace for every legacy token in scope, sample each match + to distinguish identifier-position from grammar-position uses, and record + the per-file migration list at the bottom of the Surprises section. The + Progress entry for stage A is the recorded list. +2. Add red tests under `src/parser/tests/reserved_tokens.rs` for every + reject path, and add regression cases for the preserved `as` alias and + `#[...]` attribute paths. +3. Invert the `typedef`-acceptance assertions in + `src/parser/tests/parser.rs` (lines 272–293) and adjust any shared fixtures + under `src/parser/tests/programs.rs` and `src/parser/tests/specs.rs` that + depended on them. Migrate any `examples/*.dl` fixtures that use rejected + tokens to the modern equivalents in the same commit. +4. Add `src/parser/reserved_tokens.rs` with the eight `pub(crate)` + diagnostic constants, including the contextual bare-`#` case, the + `rejection_for` predicate, and the `reserved_token_error` constructor that + yields a `Simple::custom` carrying the matching message. +5. Keep `handle_typedef` in `src/parser/span_scanners/typedefs.rs` on its + line-skipping recovery path and ensure that `collect_reserved_token_errors` + in `src/parser/span_scanner.rs` routes `K_TYPEDEF` through + `reserved_tokens::rejection_for` plus `reserved_token_error`. Remove or + redirect any helper that previously assumed `typedef` would produce a + `TypeDef` span. +6. Extend the top-level span scanner so every drawn token is classified + via `reserved_tokens::rejection_for` and rejected through the shared + constructor. Add the bare-`T_HASH` lookahead so attribute uses still reach + the attribute scanner unchanged. +7. Extend the Pratt expression layer in `src/parser/expression/` so + `T_SPACESHIP` and any legacy type-name keyword inside an expression context + routes through the same predicate and recovers to the next infix boundary. +8. Extend the type-position scanners so the five legacy type names route + through the same predicate, producing the modern-type fix hint. No site + constructs its own `Simple::custom` for a rejected reserved + token. +9. Add behavioural tests in `tests/reserved_token_rejection.rs` covering + each token's reject path plus the preserved `as` and `#[...]` uses. +10. Add the linter / sema regression test described under stage D so no + rule silently misses now-error legacy-keyword input. +11. Update active documentation (spec sections `2.3`, `5.2`, `9.1`; + conformance register item `14`; parser-implementation-notes; + ddlint-design; users' guide; developers' guide; `CHANGELOG.md` or + the closest migration-notes file). +12. Run the Make targets listed under validation and fix any failures. +13. Mark roadmap item `2.6.7` done and close the conformance register + entry as `implemented`. + +## Validation commands + +Run the following from the repository root, capturing output with `tee`: + +```shell +set -o pipefail; make fmt 2>&1 | tee /tmp/2-6-7-make-fmt.log +set -o pipefail; make check-fmt 2>&1 | tee /tmp/2-6-7-make-check-fmt.log +set -o pipefail; make lint 2>&1 | tee /tmp/2-6-7-make-lint.log +set -o pipefail; CI=1 make test 2>&1 | tee /tmp/2-6-7-make-test.log +set -o pipefail; make markdownlint 2>&1 | tee /tmp/2-6-7-make-markdownlint.log +set -o pipefail; make nixie 2>&1 | tee /tmp/2-6-7-make-nixie.log +``` + +Successful completion means all six commands exit with status `0`, the new +reserved-token unit and behavioural suites pass, the `as` alias and `#[...]` +attribute regression cases still pass, and the roadmap entry can be closed +without leaving the conformance register in `scheduled` state. + +## Interfaces and dependencies + +This plan does not introduce new external dependencies. The new internal +surface is constrained as follows. + +In `src/parser/reserved_tokens.rs`, expose only crate-local items. The message +constants and the classification predicate are `pub(crate)` because the +diagnostic strings are not a public contract; tests inside the crate import the +constants by name so any wording change forces test breakage. Use +`&'static str` throughout — every message is statically known and fits the +`Simple::custom` payload without allocation. + +```rust +pub(crate) const RESERVED_TYPEDEF_ERROR: &str = + "`typedef` is a legacy DDlog keyword; use `type` instead"; +pub(crate) const RESERVED_SPACESHIP_ERROR: &str = + "`<=>` was reserved upstream but has no semantics in DDlog; remove it"; +pub(crate) const RESERVED_BARE_HASH_ERROR: &str = + "`#` is reserved; only `#[...]` attribute syntax is accepted"; +pub(crate) const RESERVED_BIGINT_ERROR: &str = + "`bigint` is a legacy type name; use a sized integer such as `i64` or `u64`"; +pub(crate) const RESERVED_BIT_ERROR: &str = + "`bit` is a legacy type name; use an unsigned sized integer such as `u32`"; +pub(crate) const RESERVED_DOUBLE_ERROR: &str = + "`double` is a legacy type name; use `f64`"; +pub(crate) const RESERVED_FLOAT_ERROR: &str = + "`float` is a legacy type name; use `f32`"; +pub(crate) const RESERVED_SIGNED_ERROR: &str = + "`signed` is a legacy type name; use a signed sized integer such as `i32`"; + +/// Returns the matching diagnostic message when `kind` is a reserved +/// token that must be rejected by the parser, or `None` otherwise. The +/// bare-`#` case is selected by contextual lookahead: `T_LBRACKET` is the +/// attribute prefix, and `collect_reserved_token_errors`, invoked by the +/// top-level span scanner, calls `reserved_token_error` with +/// `RESERVED_BARE_HASH_ERROR` when the lookahead fails. +pub(crate) fn rejection_for( + kind: crate::SyntaxKind, +) -> Option<&'static str>; + +/// Constructs the deterministic `Simple::custom` carried by +/// `Parsed::errors`. Every enforcement site routes through this helper so +/// the message wording stays single-sourced. +pub(crate) fn reserved_token_error( + span: crate::Span, + message: &'static str, +) -> chumsky::error::Simple; +``` + +Routing every enforcement site through `rejection_for` plus +`reserved_token_error` keeps the eight messages aligned, with the bare-`#` +message selected contextually. Full-program collection is centralized in the +top-level span scanner, while expression-only parsing uses the same helper. A +constraint in the section above forbids any site from constructing its own +`Simple::custom` for a rejected reserved token. + +The preserved interfaces are: + +- `crate::parser::ast::Import::alias()` keeps returning the `as`-bound + identifier for `import X as Y` programmes. +- `crate::parser::span_scanners::attributes::*` keeps consuming `T_HASH` + `T_LBRACKET` sequences unchanged. +- `crate::Parsed::errors` keeps its `Vec>` shape; the + new diagnostics simply appear in that vector alongside existing parser + diagnostics. + +## Idempotence and recovery + +Each implementation step is a small, reviewable diff and is safe to repeat: +applying the same edit twice has no effect. If a step fails mid-way, revert the +staged changes with `git restore --staged --worktree ` and rerun from +the previous concrete step. The validation Make targets are themselves +idempotent. + +## Related documentation and skills + +A novice continuing this work should consult, in roughly this order: + +- `docs/parser-conformance-register.md` item `14` for the open delta. +- `docs/differential-datalog-parser-syntax-spec-updated.md` sections `2.3`, + `5.2`, and `9.1` for the normative grammar being changed. +- `docs/parser-implementation-notes.md` for the parser-boundary conventions + used elsewhere in the scanner stack. +- `docs/building-an-error-recovering-parser-with-chumsky.md` for the + chumsky idioms used by the existing diagnostic constructors. +- `docs/rust-parser-testing-comprehensive-guide.md` and + `docs/rust-testing-with-rstest-fixtures.md` for the test-shape conventions + expected by the existing suites. +- `docs/ddlint-design.md` for the broader parser-contract story. +- `docs/complexity-antipatterns-and-refactoring-strategies.md` if any + scanner refactor grows past the file-length threshold during stage B. + +Relevant skills the implementing agent should load: + +- `rust-router`, then `rust-errors` for the `Simple::custom` + shape and message hygiene, and `rust-types-and-apis` if the diagnostic helper + grows beyond constants and one function. +- `nextest` for the test runner conventions used by `CI=1 make test`. +- `commit-message` and `pr-creation` to land the change. + +## Revision note + +This plan was initially drafted on 2026-05-29 and immediately revised after a +Logisphere community-of-experts review on the same day. + +What changed in the post-review revision: + +- The policy module was renamed from `legacy_tokens` to + `reserved_tokens` (Pandalump): `as` is also a legacy token but is kept, so + the more precise name avoids confusion. +- A single classification predicate `rejection_for(kind)` was added and + every enforcement site now routes through it (Pandalump): the previous draft + risked message drift between full-program collection and expression parsing + paths. +- The message constants were pinned to `pub(crate)` and the messages + themselves were declared *not* a public contract (Telefono): tests import the + constants by name so any drift breaks the tests. +- The workspace audit was pre-quantified in the Surprises section + (Wafflecat): the counts confirm the scope tolerance is realistic and show that + `typedef` migration is confined to `examples/*.dl` fixtures. +- An explicit linter / sema regression test was added under stage D + (Doggylump): guards against the most likely pre-mortem scenario where a rule + silently misses now-error input. +- A `CHANGELOG.md` (or equivalent migration-notes) update was added to + the documentation step (Dinolump): adopters scanning a release note are more + likely to find the breaking changes there than in the users' guide alone. +- The Decision Log gained explicit entries recording (a) why the cheaper + lex-stage rejection was considered and rejected, (b) why the module is named + `reserved_tokens`, and (c) why `typedef` is cut over immediately rather than + via a deprecation cycle. + +Why these revisions: each addresses a 🟡 or 💡 finding from the Logisphere +review. None changes the structural choice (parse-stage rejection, named token +kinds preserved) — they sharpen the policy and its enforcement so a novice can +follow the plan to completion without relying on tacit knowledge. + +Effect on remaining work: the workspace audit step is now explicit at the top +of stage A, the predicate constraint binds every enforcement site identically, +and the test plan now commits to one extra regression test under the linter / +sema suite. The approved scope budget (≤28 files, ≤800 net LOC) remains the +limit because the new module is small and the new regression test is a single +rstest case. + +Subsequent revisions must append a short note describing what changed, why, and +how it affects the remaining work, and must update the Status field above. diff --git a/docs/parser-conformance-register.md b/docs/parser-conformance-register.md index f7a8624b..ee892bda 100644 --- a/docs/parser-conformance-register.md +++ b/docs/parser-conformance-register.md @@ -177,11 +177,14 @@ This register tracks parser behaviour against the syntax specification. ### 14. Legacy token compatibility policy - Topic: `typedef`, `as`, legacy type names, `#`, `<=>` policy completion. -- Current behaviour (code): tokenizer still recognizes these tokens and parser - treatment is mixed (`src/tokenizer.rs`). -- Spec/target behaviour: section 9.1 records compatibility intent but not a - fully closed policy matrix. -- Decision status: `scheduled`. +- Current behaviour (code): tokenizer recognizes these tokens, and + `src/parser/reserved_tokens.rs` single-sources parser diagnostics for + unsupported uses. `type`, `import X as Y`, and `#[...]` remain accepted. +- Spec/target behaviour: section 9.1 records the closed policy matrix. + `typedef`, legacy type names, bare `#`, and `<=>` are rejected with + deterministic diagnostics; `as` and attribute `#` keep their supported + grammar roles. +- Decision status: `implemented`. - Roadmap item: `docs/roadmap.md` item `2.6.7`. ### 15. Brace-group extension `{ expr }` diff --git a/docs/parser-implementation-notes.md b/docs/parser-implementation-notes.md index 569ea6db..36908c4d 100644 --- a/docs/parser-implementation-notes.md +++ b/docs/parser-implementation-notes.md @@ -323,6 +323,13 @@ Important invariants: output-signature check and emits the targeted diagnostic `transformer declarations require ':' followed by at least one output identifier` when the colon or first output identifier is missing. +- Reserved-token compatibility diagnostics are owned exclusively by + `src/parser/reserved_tokens.rs`. The lexer keeps legacy token kinds such as + `K_TYPEDEF`, `K_BIGINT`, and `T_SPACESHIP` so parser recovery can report + exact spans. This module owns the compatibility messages, + `reserved_tokens::rejection_for`, which selects a rejection message, and + `reserved_tokens::reserved_token_error`, which constructs the span-attached + diagnostic. These helpers are shared intentionally to keep declaration parsing consistent across top-level constructs. @@ -349,13 +356,16 @@ Representative diagnostics include: - malformed aggregation signatures, - invalid transformer declaration forms. -### Centralized diagnostic messages +### Feature-specific diagnostic messages -Diagnostic strings that form part of the parser's contract (asserted in tests -or exposed through `Parsed::errors()`) are defined once in +Feature-specific diagnostic strings that form part of the parser's contract +(asserted in tests or exposed through `Parsed::errors()`) are defined once in `src/parser/error_messages.rs` and re-exported via `src/test_util/mod.rs`. Scanner code and test helpers import the same constants, so message text cannot -drift between production code and assertions. +drift between production code and assertions. Reserved-token compatibility +diagnostics are excluded from this module and are exclusively owned by +`src/parser/reserved_tokens.rs`, including `rejection_for` and +`reserved_token_error`. Constants currently defined: @@ -372,7 +382,7 @@ Test utilities that match these messages: counterpart; asserts that no custom error contains `pattern`. Both helpers normalize internal token names to human-readable forms before -comparison (via `normalise_tokens`), so assertion strings can use either raw +comparison (via `normalize_tokens`), so assertion strings can use either raw token names or their human-readable equivalents. ## File index @@ -383,7 +393,8 @@ token names or their human-readable equivalents. - Pratt postfix helpers: `src/parser/expression/pratt/{postfix,diff,delay}.rs` - Prefix/infix helpers: `src/parser/expression/*.rs` -- Centralized diagnostic messages: `src/parser/error_messages.rs` +- Feature-specific diagnostic messages: `src/parser/error_messages.rs` +- Reserved-token compatibility diagnostics: `src/parser/reserved_tokens.rs` - Rule span scanning: `src/parser/span_scanners/rules.rs` - Top-level scanners: `src/parser/span_scanners/*.rs` - AST wrappers: `src/parser/ast/*.rs` diff --git a/docs/roadmap.md b/docs/roadmap.md index d709dc70..ca8fa4b2 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -228,7 +228,7 @@ split parser library surfaces defined in `docs/adr-001-parser-crate-split.md`. - [ ] 2.6.6.1. Add typed AST access for spec-form relation primary-key expressions. The parser currently preserves the expression as opaque CST text and `Relation::primary_key()` exposes only the binder/list names. -- [ ] 2.6.7. Finalize legacy token compatibility policy (`typedef`, `as`, +- [x] 2.6.7. Finalize legacy token compatibility policy (`typedef`, `as`, legacy type names, `#`, `<=>`) with deterministic diagnostics. See docs/parser-conformance-register.md item 14. - [ ] 2.6.8. Decide brace-group extension policy (`{ expr }`): codify or diff --git a/docs/users-guide.md b/docs/users-guide.md index a7d04f8a..c10f94db 100644 --- a/docs/users-guide.md +++ b/docs/users-guide.md @@ -63,6 +63,24 @@ assert!(diagnostics.is_empty()); `ddlint` uses the `log` API for parser warnings. Initialize a logger in your binary and use `RUST_LOG` to control verbosity (for example `RUST_LOG=warn`). +## Legacy DDlog tokens + +The parser keeps legacy token kinds in the lexer so diagnostics can point to +the exact source span, but unsupported legacy syntax is rejected during parsing. + +- Use `type Foo = ...` instead of `typedef Foo = ...`. +- Use sized integer types such as `i64`, `u64`, and `u32` instead of `bigint` + or `bit`. +- Use `f64` instead of `double`. +- Use `f32` instead of `float`. +- Use signed sized integer types such as `i32` instead of `signed`. +- Remove `<=>`; it was reserved upstream but has no DDlog semantics in + `ddlint`. +- Use `#[...]` for attributes. A bare `#` is rejected. + +`as` remains valid in supported grammar positions, including import aliases +such as `import foo::bar as baz`. + ## Relation declarations Relation declarations may combine an optional role, optional kind, optional diff --git a/examples/extern_transformer_decl.dl b/examples/extern_transformer_decl.dl index 0afa4722..8778263d 100644 --- a/examples/extern_transformer_decl.dl +++ b/examples/extern_transformer_decl.dl @@ -1,5 +1,5 @@ /* Extern transformer declaration (parseable without an impl) */ -typedef node_t = u64 +type node_t = u64 input relation Edge(from: node_t, to: node_t) extern transformer scc(edges: relation[(node_t, node_t)], diff --git a/examples/functions_and_match.dl b/examples/functions_and_match.dl index 97641704..34a795cc 100644 --- a/examples/functions_and_match.dl +++ b/examples/functions_and_match.dl @@ -1,8 +1,8 @@ /* Algebraic data types, Option, and pattern matching */ -typedef IPAddress = IPv4Address{ipv4addr: bit<32>} | IPv6Address{ipv6addr: bit<128>} -typedef OptionalIPAddress = Option +type IPAddress = IPv4Address{ipv4addr: u32} | IPv6Address{ipv6addr: u128} +type OptionalIPAddress = Option -function last_byte(a: OptionalIPAddress): bit<8> { +function last_byte(a: OptionalIPAddress): u8 { match (a) { None -> 0, Some{x = IPv4Address{ipv4addr = addr}} -> addr[7:0], @@ -11,8 +11,8 @@ function last_byte(a: OptionalIPAddress): bit<8> { } input relation Host(address: OptionalIPAddress) -output relation IPv6Addr(addr: bit<128>) -output relation HostLastByte(address: OptionalIPAddress, last: bit<8>) +output relation IPv6Addr(addr: u128) +output relation HostLastByte(address: OptionalIPAddress, last: u8) // Extract IPv6 addresses by matching Some{IPv6Address{addr}} and emitting addr. IPv6Addr(addr) :- Host(.address = Some{x = IPv6Address{ipv6addr = addr}}). diff --git a/examples/hello_join.dl b/examples/hello_join.dl index cac1fb7b..a270a1a3 100644 --- a/examples/hello_join.dl +++ b/examples/hello_join.dl @@ -1,5 +1,5 @@ /* Basic join over typed input relations */ -typedef Category = CategoryStarWars | CategoryOther +type Category = CategoryStarWars | CategoryOther input relation Word1(word: string, cat: Category) input relation Word2(word: string, cat: Category) diff --git a/examples/left_join_by_negation.dl b/examples/left_join_by_negation.dl index deb5adbb..69dadfa6 100644 --- a/examples/left_join_by_negation.dl +++ b/examples/left_join_by_negation.dl @@ -1,17 +1,17 @@ /* Left join semantics via negation; binding rows to variables */ -input relation Endpoint(ip: bit<32>, proto: string, preferred_port: bit<16>) +input relation Endpoint(ip: u32, proto: string, preferred_port: u16) input relation Blocklisted(endpoint: string) -function addr_port(ip: bit<32>, proto: string, preferred_port: bit<16>): string { - var port: bit<16> = match (proto) { +function addr_port(ip: u32, proto: string, preferred_port: u16): string { + var port: u16 = match (proto) { "FTP" -> 21, "HTTPS" -> 443, _ -> (if (preferred_port != 0) preferred_port else 80) }; - var o1: bit<8> = ip[31:24]; - var o2: bit<8> = ip[23:16]; - var o3: bit<8> = ip[15:8]; - var o4: bit<8> = ip[7:0]; + var o1: u8 = ip[31:24]; + var o2: u8 = ip[23:16]; + var o3: u8 = ip[15:8]; + var o4: u8 = ip[7:0]; "${o1}.${o2}.${o3}.${o4}:${port}" } diff --git a/examples/paths_excluding.dl b/examples/paths_excluding.dl index a169dcbf..872f8bb6 100644 --- a/examples/paths_excluding.dl +++ b/examples/paths_excluding.dl @@ -1,5 +1,5 @@ /* Transitive closure while excluding certain nodes */ -typedef node_t = u64 +type node_t = u64 input relation Edge(from: node_t, to: node_t) input relation Exclude(node: node_t) diff --git a/examples/primary_key_and_index.dl b/examples/primary_key_and_index.dl index 23e8f113..02690b42 100644 --- a/examples/primary_key_and_index.dl +++ b/examples/primary_key_and_index.dl @@ -1,5 +1,5 @@ /* Explicit record type relation, primary key, and a simple index */ -typedef Person = Person{ +type Person = Person{ name: string, nationality: string, occupation: string diff --git a/examples/reachability.dl b/examples/reachability.dl index a65c0b6d..1c9cb6c5 100644 --- a/examples/reachability.dl +++ b/examples/reachability.dl @@ -1,5 +1,5 @@ /* Transitive closure over a directed graph */ -typedef node_t = u64 +type node_t = u64 input relation Edge(from: node_t, to: node_t) output relation Path(src: node_t, dst: node_t) diff --git a/examples/ref_and_intern.dl b/examples/ref_and_intern.dl index 66db9b42..51e99d2d 100644 --- a/examples/ref_and_intern.dl +++ b/examples/ref_and_intern.dl @@ -1,7 +1,7 @@ /* Using reference-backed relations and interned strings */ -typedef Student = Student{id: u64, name: istring, school: istring} +type Student = Student{id: u64, name: istring, school: istring} -/* The typedef defines the record; the relation uses it via `&` syntax. */ +/* The type defines the record; the relation uses it via `&` syntax. */ /* `&Student(…)` desugars to a relation of `Ref` */ input relation &Student(id: u64, name: istring, school: istring) diff --git a/examples/tuple_destructuring.dl b/examples/tuple_destructuring.dl index c373e07c..edd95b29 100644 --- a/examples/tuple_destructuring.dl +++ b/examples/tuple_destructuring.dl @@ -1,9 +1,9 @@ /* Functions returning tuples; tuple pattern destructuring in rules */ -function bytes_of_ip(a: bit<32>): (bit<8>, bit<8>, bit<8>, bit<8>) { +function bytes_of_ip(a: u32): (u8, u8, u8, u8) { (a[31:24], a[23:16], a[15:8], a[7:0]) } -input relation Host4(addr: bit<32>) -output relation IntranetHost4(addr: bit<32>) +input relation Host4(addr: u32) +output relation IntranetHost4(addr: u32) IntranetHost4(a) :- Host4(a), (192, 168, _, _) = bytes_of_ip(a). diff --git a/src/linter/runner.rs b/src/linter/runner.rs index fb72610c..1b5ad01a 100644 --- a/src/linter/runner.rs +++ b/src/linter/runner.rs @@ -53,6 +53,7 @@ use crate::sema::SemanticModel; pub struct Runner<'a> { store: &'a CstRuleStore, green: GreenNode, + has_parse_errors: bool, source_text: Arc, semantic_model: Arc, config: RuleConfig, @@ -74,6 +75,7 @@ impl<'a> Runner<'a> { Self { store, green: parsed.green().clone(), + has_parse_errors: !parsed.errors().is_empty(), source_text: source_text.into(), semantic_model: Arc::new(crate::sema::build(parsed)), config, @@ -82,11 +84,14 @@ impl<'a> Runner<'a> { /// Execute all registered rules against the CST in parallel. /// + /// Returns no diagnostics when parsing failed, because recovery CST nodes + /// must not trigger lint rules. + /// /// Returns diagnostics sorted by span start, then span end, then rule /// name, ensuring deterministic output regardless of thread scheduling. #[must_use] pub fn run(&self) -> Vec { - if self.store.is_empty() { + if self.has_parse_errors || self.store.is_empty() { return Vec::new(); } diff --git a/src/parser/ast/type_def.rs b/src/parser/ast/type_def.rs index 4443b226..ca2fa4ee 100644 --- a/src/parser/ast/type_def.rs +++ b/src/parser/ast/type_def.rs @@ -1,14 +1,14 @@ //! //! AST wrapper for type definitions in `DDlog`. //! -//! This module exposes the `TypeDef` struct for both regular `typedef` and +//! This module exposes the `TypeDef` struct for both regular `type` and //! `extern type` declarations. It enables extraction of the type name, and //! whether the declaration is marked as `extern`. use super::AstNode; use crate::{DdlogLanguage, SyntaxKind}; -/// Typed wrapper for a `typedef` or `extern type` declaration. +/// Typed wrapper for a `type` or `extern type` declaration. #[derive(Debug, Clone)] pub struct TypeDef { pub(crate) syntax: rowan::SyntaxNode, @@ -59,14 +59,14 @@ mod tests { .type_defs() .first() .cloned() - .expect("typedef missing"); + .expect("type missing"); assert_eq!(td.name().as_deref(), Some("Handle")); assert!(td.is_extern()); } #[test] - fn regular_typedef_parsed() { - let parsed = parse("typedef UserId = u64"); + fn regular_type_parsed() { + let parsed = parse("type UserId = u64"); crate::test_util::assert_no_parse_errors(parsed.errors()); #[expect(clippy::expect_used, reason = "Using expect for clearer test failures")] let td = parsed @@ -74,14 +74,14 @@ mod tests { .type_defs() .first() .cloned() - .expect("typedef missing"); + .expect("type missing"); assert_eq!(td.name().as_deref(), Some("UserId")); assert!(!td.is_extern()); } #[test] - fn typedef_name_span_points_to_declaration_identifier() { - let source = "typedef UserId = UserId"; + fn type_name_span_points_to_declaration_identifier() { + let source = "type UserId = UserId"; let parsed = parse(source); crate::test_util::assert_no_parse_errors(parsed.errors()); #[expect(clippy::expect_used, reason = "Using expect for clearer test failures")] @@ -90,13 +90,13 @@ mod tests { .type_defs() .first() .cloned() - .expect("typedef missing"); + .expect("type missing"); let span = td .name_span() - .unwrap_or_else(|| panic!("missing typedef name_span in `{source}`")); + .unwrap_or_else(|| panic!("missing type name_span in `{source}`")); assert_eq!(span_text(source, &span), "UserId"); - assert_eq!(span.start, "typedef ".len()); + assert_eq!(span.start, "type ".len()); } } diff --git a/src/parser/cst_builder/tree.rs b/src/parser/cst_builder/tree.rs index 01ebda44..475a1ae6 100644 --- a/src/parser/cst_builder/tree.rs +++ b/src/parser/cst_builder/tree.rs @@ -234,7 +234,7 @@ mod tests { fn build_green_tree_matches_span_positions() { let src = concat!( "import foo::bar\n", - "typedef UserId = u64\n", + "type UserId = u64\n", "input relation User(id: UserId, name: string) primary key (id)\n", "index Idx_User_name(name: string) on User[name]\n", "function greet(name: string): string {\n", diff --git a/src/parser/expression/infix.rs b/src/parser/expression/infix.rs index ddbac611..41019af4 100644 --- a/src/parser/expression/infix.rs +++ b/src/parser/expression/infix.rs @@ -1,6 +1,7 @@ //! Infix operator handling for the Pratt parser. use crate::parser::ast::{Expr, infix_binding_power}; +use crate::parser::reserved_tokens::rejection_for; use crate::{Span, SyntaxKind}; use super::pratt::Pratt; @@ -30,6 +31,14 @@ where break; } + if let Some(message) = rejection_for(op_kind) { + let Some((_, op_span)) = self.ts.next_tok() else { + unreachable!("peeked reserved token"); + }; + self.ts.push_reserved_error(op_span, message); + return None; + } + let Some((l_bp, r_bp, op)) = infix_binding_power(op_kind) else { break; }; diff --git a/src/parser/expression/pratt.rs b/src/parser/expression/pratt.rs index 8059713b..535c6f94 100644 --- a/src/parser/expression/pratt.rs +++ b/src/parser/expression/pratt.rs @@ -9,6 +9,7 @@ use std::collections::HashMap; use chumsky::error::Simple; use crate::parser::ast::{Expr, StringLiteral}; +use crate::parser::reserved_tokens::rejection_for; use crate::{Span, SyntaxKind, tokenize_without_trivia}; use super::token_stream::TokenStream; @@ -91,9 +92,13 @@ pub fn parse_expression(src: &str) -> Result>> { let expr = parser.parse_expr(0); if let Some(expr_val) = expr { for (kind, sp) in parser.ts.drain_unexpected_tokens() { - parser - .ts - .push_error(sp, format!("unexpected token: {kind:?}")); + if let Some(message) = rejection_for(kind) { + parser.ts.push_reserved_error(sp, message); + } else { + parser + .ts + .push_error(sp, format!("unexpected token: {kind:?}")); + } } if !parser.ts.has_errors() { return Ok(expr_val); diff --git a/src/parser/expression/prefix.rs b/src/parser/expression/prefix.rs index 9878407a..66b1ebde 100644 --- a/src/parser/expression/prefix.rs +++ b/src/parser/expression/prefix.rs @@ -1,6 +1,7 @@ //! Parsing of prefix expressions for the Pratt parser. use crate::parser::ast::{Expr, prefix_binding_power}; +use crate::parser::reserved_tokens::rejection_for; use crate::{Span, SyntaxKind}; use super::pratt::Pratt; @@ -36,6 +37,10 @@ where SyntaxKind::K_CONTINUE => Some(Self::parse_continue_expression()), SyntaxKind::K_RETURN => self.parse_return_expression(), k => { + if let Some(message) = rejection_for(k) { + self.ts.push_reserved_error(span.clone(), message); + return None; + } let Some((bp, op)) = prefix_binding_power(k) else { self.ts .push_error(span.clone(), format!("unexpected token: {k:?}")); diff --git a/src/parser/expression/token_stream.rs b/src/parser/expression/token_stream.rs index 34e6b977..26ded6ed 100644 --- a/src/parser/expression/token_stream.rs +++ b/src/parser/expression/token_stream.rs @@ -7,6 +7,7 @@ use std::iter::Peekable; use chumsky::error::Simple; +use crate::parser::reserved_tokens::reserved_token_error; use crate::{Span, SyntaxKind}; pub(super) struct TokenStream<'a, I> @@ -94,6 +95,10 @@ where self.errors.push(Simple::custom(span, msg.into())); } + pub(super) fn push_reserved_error(&mut self, span: Span, msg: &'static str) { + self.errors.push(reserved_token_error(span, msg)); + } + pub(super) fn slice(&self, span: &Span) -> String { debug_assert!(span.end <= self.src.len(), "lexer produced invalid span"); self.src.get(span.clone()).unwrap_or("").to_string() diff --git a/src/parser/mod.rs b/src/parser/mod.rs index 2d3df3e3..fc4b7076 100644 --- a/src/parser/mod.rs +++ b/src/parser/mod.rs @@ -14,6 +14,7 @@ mod lexer_helpers; mod token_stream; pub(crate) mod error_messages; +pub(crate) mod reserved_tokens; pub(crate) mod span_utils; mod span_collector; diff --git a/src/parser/reserved_tokens.rs b/src/parser/reserved_tokens.rs new file mode 100644 index 00000000..259c42d7 --- /dev/null +++ b/src/parser/reserved_tokens.rs @@ -0,0 +1,102 @@ +//! Diagnostics for tokens reserved without parser semantics. +//! +//! The active syntax specification section 9.1 keeps these token kinds in the +//! lexer for precise spans, but rejects unsupported uses in the parser with a +//! deterministic message and fix hint. + +use std::collections::HashSet; + +use chumsky::error::{Simple, SimpleReason}; + +use crate::{Span, SyntaxKind}; + +pub(crate) const RESERVED_TYPEDEF_ERROR: &str = + "`typedef` is a legacy DDlog keyword; use `type` instead"; +pub(crate) const RESERVED_SPACESHIP_ERROR: &str = + "`<=>` was reserved upstream but has no semantics in DDlog; remove it"; +pub(crate) const RESERVED_BARE_HASH_ERROR: &str = + "`#` is reserved; only `#[...]` attribute syntax is accepted"; +pub(crate) const RESERVED_BIGINT_ERROR: &str = + "`bigint` is a legacy type name; use a sized integer such as `i64` or `u64`"; +pub(crate) const RESERVED_BIT_ERROR: &str = + "`bit` is a legacy type name; use an unsigned sized integer such as `u32`"; +pub(crate) const RESERVED_DOUBLE_ERROR: &str = "`double` is a legacy type name; use `f64`"; +pub(crate) const RESERVED_FLOAT_ERROR: &str = "`float` is a legacy type name; use `f32`"; +pub(crate) const RESERVED_SIGNED_ERROR: &str = + "`signed` is a legacy type name; use a signed sized integer such as `i32`"; + +pub(crate) fn rejection_for(kind: SyntaxKind) -> Option<&'static str> { + match kind { + SyntaxKind::K_TYPEDEF => Some(RESERVED_TYPEDEF_ERROR), + SyntaxKind::T_SPACESHIP => Some(RESERVED_SPACESHIP_ERROR), + SyntaxKind::K_BIGINT => Some(RESERVED_BIGINT_ERROR), + SyntaxKind::K_BIT => Some(RESERVED_BIT_ERROR), + SyntaxKind::K_DOUBLE => Some(RESERVED_DOUBLE_ERROR), + SyntaxKind::K_FLOAT => Some(RESERVED_FLOAT_ERROR), + SyntaxKind::K_SIGNED => Some(RESERVED_SIGNED_ERROR), + _ => None, + } +} + +pub(crate) fn reserved_token_error(span: Span, message: &'static str) -> Simple { + Simple::custom(span, message) +} + +pub(crate) fn collect_reserved_token_errors( + tokens: &[(SyntaxKind, Span)], + parse_errors: &[Simple], +) -> Vec> { + let emitted_reserved_errors = emitted_reserved_errors(parse_errors); + tokens + .iter() + .enumerate() + .filter_map(|(idx, (kind, span))| { + let message = reserved_message_for_token(tokens, idx, *kind)?; + (!emitted_reserved_errors.contains(&(span.clone(), message))) + .then(|| reserved_token_error(span.clone(), message)) + }) + .collect() +} + +fn emitted_reserved_errors(errors: &[Simple]) -> HashSet<(Span, &'static str)> { + errors + .iter() + .filter_map(|error| match error.reason() { + SimpleReason::Custom(message) => { + reserved_message(message).map(|message| (error.span().clone(), message)) + } + _ => None, + }) + .collect() +} + +fn reserved_message(message: &str) -> Option<&'static str> { + [ + RESERVED_TYPEDEF_ERROR, + RESERVED_SPACESHIP_ERROR, + RESERVED_BARE_HASH_ERROR, + RESERVED_BIGINT_ERROR, + RESERVED_BIT_ERROR, + RESERVED_DOUBLE_ERROR, + RESERVED_FLOAT_ERROR, + RESERVED_SIGNED_ERROR, + ] + .into_iter() + .find(|candidate| *candidate == message) +} + +fn reserved_message_for_token( + tokens: &[(SyntaxKind, Span)], + idx: usize, + kind: SyntaxKind, +) -> Option<&'static str> { + if kind == SyntaxKind::T_HASH { + return is_bare_hash(tokens, idx).then_some(RESERVED_BARE_HASH_ERROR); + } + + rejection_for(kind) +} + +fn is_bare_hash(tokens: &[(SyntaxKind, Span)], idx: usize) -> bool { + !matches!(tokens.get(idx + 1), Some((SyntaxKind::T_LBRACKET, _))) +} diff --git a/src/parser/span_scanner.rs b/src/parser/span_scanner.rs index 42533016..100b9b59 100644 --- a/src/parser/span_scanner.rs +++ b/src/parser/span_scanner.rs @@ -8,6 +8,7 @@ use crate::{Span, SyntaxKind}; use super::ParsedSpans; +use super::reserved_tokens::collect_reserved_token_errors; use super::span_scanners::{ collect_apply_spans, collect_attribute_spans, collect_function_spans, collect_import_spans, collect_index_spans, collect_relation_spans, collect_rule_spans, collect_transformer_spans, @@ -49,6 +50,7 @@ pub(super) fn parse_tokens( let non_rule_spans = merge_spans(non_rule_spans); let (rule_spans, expr_spans, rule_errors) = collect_rule_spans(tokens, src, &non_rule_spans); + let reserved_errors = collect_reserved_token_errors(tokens, &rule_errors); let mut all_errors = attribute_errors; all_errors.extend(errors); @@ -60,6 +62,7 @@ pub(super) fn parse_tokens( all_errors.extend(apply_errors); all_errors.extend(rule_errors); all_errors.extend(lexer_errors(tokens)); + all_errors.extend(reserved_errors); let span_result = ParsedSpans::builder() .attributes(attribute_spans) diff --git a/src/parser/span_scanners/attributes.rs b/src/parser/span_scanners/attributes.rs index 8991241f..2b66fadc 100644 --- a/src/parser/span_scanners/attributes.rs +++ b/src/parser/span_scanners/attributes.rs @@ -1,7 +1,7 @@ //! Scanner for attribute spans and placement validation. //! //! Detects `#[…]` attribute syntax, records attribute spans, and validates -//! that attributes precede only permitted item kinds (typedef, function, +//! that attributes precede only permitted item kinds (type, function, //! relation). use chumsky::error::Simple; @@ -13,7 +13,7 @@ type State<'a> = SpanCollector<'a, Vec>>; /// Whether the given keyword may directly begin an attributed item. /// -/// Permitted targets per spec §5.1: `Typedef`, `Function`, and +/// Permitted targets per spec §5.1: `TypeDef`, `Function`, and /// `RelationDecl`. Relation declarations may be prefixed with role keywords /// (`input`, `output`, `stream`, `multiset`). /// @@ -23,8 +23,7 @@ type State<'a> = SpanCollector<'a, Vec>>; fn is_simple_attribute_target(kind: SyntaxKind) -> bool { matches!( kind, - SyntaxKind::K_TYPEDEF - | SyntaxKind::K_TYPE + SyntaxKind::K_TYPE | SyntaxKind::K_FUNCTION | SyntaxKind::K_INPUT | SyntaxKind::K_OUTPUT @@ -73,9 +72,6 @@ fn consume_attribute(st: &mut State<'_>) -> Option { let start = hash_span.start; st.stream.advance(); // consume T_HASH - // Skip optional inline whitespace between # and [ - st.stream.skip_ws_inline(); - // Must be followed by T_LBRACKET to form an attribute let next = st.stream.peek()?; if next.0 != SyntaxKind::T_LBRACKET { diff --git a/src/parser/span_scanners/tests/attribute_tests.rs b/src/parser/span_scanners/tests/attribute_tests.rs index bc7fb371..fbb62114 100644 --- a/src/parser/span_scanners/tests/attribute_tests.rs +++ b/src/parser/span_scanners/tests/attribute_tests.rs @@ -12,7 +12,7 @@ use rstest::rstest; #[case("#[hot]\nstream R(x: u32)\n", None)] #[case("#[hot]\nmultiset R(x: u32)\n", None)] #[case("#[cold]\nextern function f()\n", None)] -#[case("#[cold]\ntypedef T = u32\n", Some("#[cold]"))] +#[case("#[cold]\ntype T = u32\n", Some("#[cold]"))] fn collect_attribute_spans_valid_on_permitted_item( #[case] src: &str, #[case] expected_text: Option<&str>, @@ -49,7 +49,7 @@ fn collect_attribute_spans_rejected_on_forbidden_item(#[case] src: &str) { #[test] fn collect_attribute_spans_stacked() { - let src = "#[a]\n#[b]\ntypedef T = u32\n"; + let src = "#[a]\n#[b]\ntype T = u32\n"; let tokens = tokenize(src); let (spans, errors) = collect_attribute_spans(&tokens, src); assert_eq!(spans.len(), 2); @@ -72,9 +72,11 @@ fn collect_attribute_spans_unclosed_bracket() { assert!(has_unclosed_error, "unexpected errors: {errors:?}"); } -#[test] -fn collect_attribute_spans_hash_without_bracket() { - let src = "# typedef T = u32\n"; +#[rstest] +#[case("# type T = u32\n")] +#[case("# [cold]\ntype T = u32\n")] +#[case("#/*comment*/[cold]\ntype T = u32\n")] +fn collect_attribute_spans_rejects_bare_hashes(#[case] src: &str) { let tokens = tokenize(src); let (spans, errors) = collect_attribute_spans(&tokens, src); assert!(spans.is_empty()); diff --git a/src/parser/span_scanners/tests/mod.rs b/src/parser/span_scanners/tests/mod.rs index a789b92b..bb607399 100644 --- a/src/parser/span_scanners/tests/mod.rs +++ b/src/parser/span_scanners/tests/mod.rs @@ -343,7 +343,7 @@ fn rule_treated_as_line_start_after_newline_trivia() { fn parse_tokens_skips_non_rule_constructs_when_scanning_rules() { let src = concat!( "import foo::bar\n", - "typedef T = string\n", + "type T = string\n", "input relation Log(id: u32) primary key (id)\n", "index I_Log(id: u32) on Log[id]\n", "function f(): u32 { return 1; }\n", diff --git a/src/parser/span_scanners/typedefs.rs b/src/parser/span_scanners/typedefs.rs index 02990dde..8ed112ce 100644 --- a/src/parser/span_scanners/typedefs.rs +++ b/src/parser/span_scanners/typedefs.rs @@ -14,12 +14,16 @@ pub(crate) fn collect_typedef_spans( ) -> (Vec, Vec>) { type State<'a> = SpanCollector<'a, Vec>>; - fn handle_typedef(st: &mut State<'_>, span: Span) { + fn handle_type(st: &mut State<'_>, span: Span) { let start = span.start; st.stream.advance(); st.push_line_span(start); } + fn handle_typedef(st: &mut State<'_>, _span: Span) { + st.skip_line(); + } + fn handle_extern(st: &mut State<'_>, span: Span) { let start = span.start; st.stream.advance(); @@ -40,6 +44,7 @@ pub(crate) fn collect_typedef_spans( token_dispatch!(st, { SyntaxKind::K_TYPEDEF => handle_typedef, + SyntaxKind::K_TYPE => handle_type, SyntaxKind::K_EXTERN => handle_extern, }); diff --git a/src/parser/tests/attributes.rs b/src/parser/tests/attributes.rs index e8d77b4c..407f9ee3 100644 --- a/src/parser/tests/attributes.rs +++ b/src/parser/tests/attributes.rs @@ -9,7 +9,7 @@ use crate::test_util::assert_no_parse_errors; use rstest::rstest; #[rstest] -#[case("#[cold]\ntypedef T = u32")] +#[case("#[cold]\ntype T = u32")] #[case("#[inline]\nfunction f() {}")] #[case("#[hot]\ninput relation R(x: u32)")] #[case("#[cold]\noutput relation R(x: u32)")] @@ -44,8 +44,8 @@ fn attribute_on_forbidden_item_emits_error(#[case] src: &str, #[case] expected_m } #[test] -fn stacked_attributes_on_typedef_no_error() { - let src = "#[a]\n#[b]\ntypedef T = u32"; +fn stacked_attributes_on_type_no_error() { + let src = "#[a]\n#[b]\ntype T = u32"; let parsed = parse(src); assert_no_parse_errors(parsed.errors()); } diff --git a/src/parser/tests/cst_integration.rs b/src/parser/tests/cst_integration.rs index 72e676bd..c3c0612b 100644 --- a/src/parser/tests/cst_integration.rs +++ b/src/parser/tests/cst_integration.rs @@ -9,7 +9,7 @@ use super::helpers::{count_nodes_by_kind, pretty_print}; fn parse_builds_cst_for_all_top_level_categories() { let src = concat!( "import foo::bar\n", - "typedef UserId = u64\n", + "type UserId = u64\n", "input relation User(id: UserId, name: string) primary key (id)\n", "index Idx_User_name(name: string) on User[name]\n", "function greet(name: string): string {\n", diff --git a/src/parser/tests/helpers.rs b/src/parser/tests/helpers.rs index ebd04a28..156b2cf3 100644 --- a/src/parser/tests/helpers.rs +++ b/src/parser/tests/helpers.rs @@ -126,7 +126,6 @@ pub(super) fn round_trip(src: impl Into) { /// /// The helper asserts that parsing succeeds without errors and that the /// extractor yields at least one item. -#[expect(clippy::expect_used, reason = "helpers used only in tests")] fn parse_single_item Vec>( src: impl Into, extractor: F, @@ -136,7 +135,10 @@ fn parse_single_item Vec>( crate::test_util::assert_no_parse_errors(parsed.errors()); assert_eq!(parsed.root().kind(), SyntaxKind::N_DATALOG_PROGRAM); let items = extractor(parsed.root()); - items.first().cloned().expect("item missing") + let Some(item) = items.first().cloned() else { + panic!("item missing"); + }; + item } /// Parse a program containing a single relation and return it. diff --git a/src/parser/tests/mod.rs b/src/parser/tests/mod.rs index c34ffdb2..e279d263 100644 --- a/src/parser/tests/mod.rs +++ b/src/parser/tests/mod.rs @@ -16,6 +16,7 @@ mod numeric_literals; mod operator_precedence; mod relation_proptest; mod relations; +mod reserved_tokens; mod round_trip; mod rules; mod transformers; diff --git a/src/parser/tests/parser.rs b/src/parser/tests/parser.rs index f1801d9f..37c21d1d 100644 --- a/src/parser/tests/parser.rs +++ b/src/parser/tests/parser.rs @@ -4,7 +4,7 @@ //! round-trips through `pretty_print` unchanged. use crate::{SyntaxKind, ast::AstNode}; -use crate::test_util::normalise_tokens; +use crate::test_util::normalize_tokens; use rstest::rstest; use super::helpers::{parse_err, parse_ok, round_trip}; @@ -152,7 +152,7 @@ fn import_missing_path() { assert!( error .expected() - .filter_map(|e| e.as_ref().map(|k| normalise_tokens(&format!("{k:?}")))) + .filter_map(|e| e.as_ref().map(|k| normalize_tokens(&format!("{k:?}")))) .any(|k| k == "identifier") ); assert_eq!(error.found(), Some(&SyntaxKind::K_AS)); @@ -160,25 +160,25 @@ fn import_missing_path() { } #[rstest] -#[case("typedef Uuid = string\n", ("Uuid", false))] -#[case("typedef UserRecord = (name: string, age: u64, active: bool)\n", ("UserRecord", false))] +#[case("type Uuid = string\n", ("Uuid", false))] +#[case("type UserRecord = (name: string, age: u64, active: bool)\n", ("UserRecord", false))] #[case("extern type FfiHandle\n", ("FfiHandle", true))] -fn typedef_parsing(#[case] src: &str, #[case] expect: (&str, bool)) { +fn type_and_extern_type_parsing(#[case] src: &str, #[case] expect: (&str, bool)) { let parsed = parse_ok(src); - assert_eq!(parsed.root().type_defs().len(), 1, "expected exactly one typedef"); + assert_eq!(parsed.root().type_defs().len(), 1, "expected exactly one type"); let def = parsed .root() .type_defs() .first() - .expect("typedef not found"); + .expect("type not found"); assert_eq!(def.name(), Some(expect.0.into())); assert_eq!(def.is_extern(), expect.1); } #[rstest] -#[case("typedef = string\n")] -#[case("typedef MissingType\n")] -fn typedef_errors(#[case] src: &str) { +#[case("type = string\n")] +#[case("type MissingType\n")] +fn type_definition_errors(#[case] src: &str) { let parsed = parse_err(src); assert!(parsed.root().type_defs().is_empty()); } diff --git a/src/parser/tests/reserved_tokens.rs b/src/parser/tests/reserved_tokens.rs new file mode 100644 index 00000000..1612e9ed --- /dev/null +++ b/src/parser/tests/reserved_tokens.rs @@ -0,0 +1,205 @@ +//! Reserved-token policy tests. +//! +//! These tests pin the parser-facing diagnostics for legacy `DDlog` tokens whose +//! lexer kinds remain available only so recovery can report precise spans. + +use chumsky::error::{Simple, SimpleReason}; +use rstest::rstest; +use std::ops::Range; + +use crate::SyntaxKind; +use crate::parser::expression::parse_expression; +use crate::parser::reserved_tokens::{ + RESERVED_BARE_HASH_ERROR, RESERVED_BIGINT_ERROR, RESERVED_BIT_ERROR, RESERVED_DOUBLE_ERROR, + RESERVED_FLOAT_ERROR, RESERVED_SIGNED_ERROR, RESERVED_SPACESHIP_ERROR, RESERVED_TYPEDEF_ERROR, +}; +use crate::test_util::assert_no_parse_errors; + +use super::helpers::{count_nodes_by_kind, parse_err, parse_ok}; + +#[test] +fn typedef_keyword_is_rejected_without_type_def_ast() { + let source = "typedef Foo = u32\n"; + let parsed = parse_err(source); + + assert!(parsed.root().type_defs().is_empty()); + assert_reserved_token_error( + parsed.errors(), + RESERVED_TYPEDEF_ERROR, + token_span(source, "typedef"), + ); +} + +#[test] +fn modern_type_keyword_still_parses_type_definitions() { + let parsed = parse_ok("type Foo = u32\n"); + + let defs = parsed.root().type_defs(); + assert_eq!(defs.len(), 1); + assert_eq!( + defs.first().and_then(crate::ast::TypeDef::name).as_deref(), + Some("Foo") + ); +} + +#[rstest] +#[case::bigint("type Foo = bigint;\n", "bigint", RESERVED_BIGINT_ERROR)] +#[case::bit("type Foo = bit<32>;\n", "bit", RESERVED_BIT_ERROR)] +#[case::double("type Foo = double;\n", "double", RESERVED_DOUBLE_ERROR)] +#[case::float("type Foo = float;\n", "float", RESERVED_FLOAT_ERROR)] +#[case::signed("type Foo = signed<32>;\n", "signed", RESERVED_SIGNED_ERROR)] +fn legacy_type_names_are_rejected_in_type_position( + #[case] src: &str, + #[case] token: &str, + #[case] expected: &str, +) { + let parsed = parse_err(src); + + assert_reserved_token_error(parsed.errors(), expected, token_span(src, token)); +} + +#[rstest] +#[case::bigint("bigint", RESERVED_BIGINT_ERROR)] +#[case::bit("bit", RESERVED_BIT_ERROR)] +#[case::double("double", RESERVED_DOUBLE_ERROR)] +#[case::float("float", RESERVED_FLOAT_ERROR)] +#[case::signed("signed", RESERVED_SIGNED_ERROR)] +fn legacy_type_names_are_rejected_outwith_type_position( + #[case] legacy_type: &str, + #[case] expected: &str, +) { + let source = format!("Output(x) :- Source(x), var {legacy_type} = x.\n"); + let expected_span = token_span(&source, legacy_type); + let parsed = parse_err(source); + + assert_reserved_token_error(parsed.errors(), expected, expected_span); +} + +#[test] +fn spaceship_operator_is_rejected_in_expression_context() { + let Err(errors) = parse_expression("1 <=> 2") else { + panic!("spaceship expression should be rejected"); + }; + + assert_reserved_token_error(&errors, RESERVED_SPACESHIP_ERROR, 2..5); +} + +#[rstest] +#[case("# foo\n")] +#[case("# [attribute]\ntype Foo = u32\n")] +#[case("#/*comment*/[attribute]\ntype Foo = u32\n")] +fn bare_hash_is_rejected(#[case] source: &str) { + let bare = parse_err(source); + assert_reserved_token_error(bare.errors(), RESERVED_BARE_HASH_ERROR, 0..1); +} + +#[test] +fn attribute_hash_is_preserved() { + let attributed = parse_ok("#[attribute]\ntype Foo = u32\n"); + assert_eq!( + count_nodes_by_kind(attributed.root().syntax(), SyntaxKind::N_ATTRIBUTE), + 1 + ); + assert_eq!(attributed.root().type_defs().len(), 1); +} + +#[test] +fn emitted_reserved_errors_are_not_duplicated() { + let source = "Output(x) :- Source(x), bigint.\n"; + let parsed = parse_err(source); + + assert_eq!( + count_custom_parse_errors(parsed.errors(), RESERVED_BIGINT_ERROR), + 1, + ); + assert_reserved_token_error( + parsed.errors(), + RESERVED_BIGINT_ERROR, + token_span(source, "bigint"), + ); +} + +#[test] +fn spaceship_in_assignment_pattern_is_rejected() { + let source = "Output(x) :- x <=> y = z.\n"; + let parsed = parse_err(source); + + assert_eq!( + count_custom_parse_errors(parsed.errors(), RESERVED_SPACESHIP_ERROR), + 1, + ); + assert_reserved_token_error( + parsed.errors(), + RESERVED_SPACESHIP_ERROR, + token_span(source, "<=>"), + ); +} + +#[test] +fn many_reserved_operators_report_once_each() { + const EXPRESSION_COUNT: usize = 128; + let body = (0..EXPRESSION_COUNT) + .map(|index| format!("value_{index} <=> value_{index}")) + .collect::>() + .join(", "); + let source = format!("Output(x) :- {body}.\n"); + let expected_spans = source + .match_indices("<=>") + .map(|(start, token)| start..start + token.len()) + .collect::>(); + let parsed = parse_err(source); + + assert_eq!( + count_custom_parse_errors(parsed.errors(), RESERVED_SPACESHIP_ERROR), + EXPRESSION_COUNT, + ); + assert_eq!( + reserved_token_spans(parsed.errors(), RESERVED_SPACESHIP_ERROR), + expected_spans, + ); +} + +fn count_custom_parse_errors(errors: &[Simple], expected: &str) -> usize { + reserved_token_spans(errors, expected).len() +} + +fn assert_reserved_token_error(errors: &[Simple], expected: &str, span: Range) { + assert_eq!(reserved_token_spans(errors, expected), vec![span]); +} + +fn reserved_token_spans(errors: &[Simple], expected: &str) -> Vec> { + errors + .iter() + .filter( + |error| matches!(error.reason(), SimpleReason::Custom(message) if message == expected), + ) + .map(|error| error.span().clone()) + .collect() +} + +fn token_span(source: &str, token: &str) -> Range { + let Some(start) = source.find(token) else { + panic!("missing token {token:?} in source {source:?}"); + }; + start..start + token.len() +} + +#[test] +fn import_alias_keyword_is_preserved() { + let parsed = parse_ok("import foo::bar as baz\n"); + let imports = parsed.root().imports(); + + assert_eq!(imports.len(), 1); + assert_eq!( + imports.first().map(crate::ast::Import::path).as_deref(), + Some("foo::bar") + ); + assert_eq!( + imports + .first() + .and_then(crate::ast::Import::alias) + .as_deref(), + Some("baz") + ); + assert_no_parse_errors(parsed.errors()); +} diff --git a/src/parser/tests/rules/aggregations.rs b/src/parser/tests/rules/aggregations.rs index ec04c7b2..bc3a185d 100644 --- a/src/parser/tests/rules/aggregations.rs +++ b/src/parser/tests/rules/aggregations.rs @@ -12,13 +12,9 @@ use crate::test_util::{ /// Assert that `body_terms()` reports an expected error for a literal found in `src`. fn assert_body_terms_error(src: &str, literal: &str, expected_error: &str) { let parsed = parse_ok(src); - #[expect(clippy::expect_used, reason = "tests require a single rule")] - let rule = parsed - .root() - .rules() - .first() - .cloned() - .expect("rule missing"); + let Some(rule) = parsed.root().rules().first().cloned() else { + panic!("rule missing"); + }; let errors = match rule.body_terms() { Ok(terms) => panic!("expected body_terms error, got {terms:?}"), Err(errs) => errs, diff --git a/src/parser/tests/rules/body_terms.rs b/src/parser/tests/rules/body_terms.rs index 2d607006..208e246d 100644 --- a/src/parser/tests/rules/body_terms.rs +++ b/src/parser/tests/rules/body_terms.rs @@ -4,22 +4,15 @@ use super::super::helpers::{parse_err, parse_ok}; use crate::parser::ast::{Expr, Pattern, RuleBodyTerm}; use crate::test_util::{call, var}; -#[expect( - clippy::expect_used, - reason = "tests require a single parsed rule for assignment assertions" -)] fn assert_body_assignment( src: &str, expected_terms_count: usize, assignment_index: usize, ) -> (Pattern, Expr) { let parsed = parse_ok(src); - let rule = parsed - .root() - .rules() - .first() - .cloned() - .expect("rule missing"); + let Some(rule) = parsed.root().rules().first().cloned() else { + panic!("rule missing"); + }; let terms = match rule.body_terms() { Ok(terms) => terms, Err(errs) => panic!("body terms should parse: {errs:?}"), diff --git a/src/parser/tests/transformers.rs b/src/parser/tests/transformers.rs index e107155c..f8d47877 100644 --- a/src/parser/tests/transformers.rs +++ b/src/parser/tests/transformers.rs @@ -208,11 +208,11 @@ fn transformer_requires_extern_for_malformed(transformer_non_extern_malformed: & let keyword_len = "transformer".len(); let expected = ErrorPattern::from("transformer declarations must be extern"); let expected_pattern = match &expected { - ErrorPattern::Custom(msg) => crate::test_util::normalise_tokens(msg), + ErrorPattern::Custom(msg) => crate::test_util::normalize_tokens(msg), }; let matching_error = errors.iter().find(|error| { let rendered = format!("{error:?}"); - let rendered_normalised = crate::test_util::normalise_tokens(&rendered); + let rendered_normalised = crate::test_util::normalize_tokens(&rendered); rendered_normalised.contains(&expected_pattern) }); let Some(error) = matching_error else { diff --git a/src/parser/tests/types.rs b/src/parser/tests/types.rs index efca4190..af08b52d 100644 --- a/src/parser/tests/types.rs +++ b/src/parser/tests/types.rs @@ -1,6 +1,6 @@ //! Type definition parsing tests. //! -//! Exercises typedef and extern type handling. +//! Exercises type and extern type handling. use super::helpers::pretty_print; use crate::parse; @@ -9,27 +9,27 @@ use crate::test_util::assert_no_parse_errors; use rstest::rstest; #[rstest] -fn standard_typedef() { - let src = "typedef Uuid = string\n"; +fn standard_type_definition() { + let src = "type Uuid = string\n"; let parsed = parse(src); assert_no_parse_errors(parsed.errors()); let defs = parsed.root().type_defs(); assert_eq!(defs.len(), 1); - #[expect(clippy::expect_used, reason = "tests require a typedef")] - let def = defs.first().expect("typedef not found"); + #[expect(clippy::expect_used, reason = "tests require a type")] + let def = defs.first().expect("type not found"); assert_eq!(def.name().as_deref(), Some("Uuid")); assert!(!def.is_extern()); } #[rstest] -fn complex_typedef() { - let src = "typedef UserRecord = (name: string, age: u64, active: bool)\n"; +fn complex_type_definition() { + let src = "type UserRecord = (name: string, age: u64, active: bool)\n"; let parsed = parse(src); assert_no_parse_errors(parsed.errors()); let defs = parsed.root().type_defs(); assert_eq!(defs.len(), 1); - #[expect(clippy::expect_used, reason = "tests require a typedef")] - let def = defs.first().expect("typedef not found"); + #[expect(clippy::expect_used, reason = "tests require a type")] + let def = defs.first().expect("type not found"); assert_eq!(def.name().as_deref(), Some("UserRecord")); assert!(!def.is_extern()); } @@ -41,8 +41,8 @@ fn extern_type() { assert_no_parse_errors(parsed.errors()); let defs = parsed.root().type_defs(); assert_eq!(defs.len(), 1); - #[expect(clippy::expect_used, reason = "tests require a typedef")] - let def = defs.first().expect("typedef not found"); + #[expect(clippy::expect_used, reason = "tests require a type")] + let def = defs.first().expect("type not found"); assert_eq!(def.name().as_deref(), Some("FfiHandle")); assert!(def.is_extern()); } @@ -57,17 +57,17 @@ fn extern_without_type_is_ignored() { } #[rstest] -#[case("typedef Uuid = string\n", "Uuid", false, "typedef Uuid = string\n")] -#[case("typedef Foo=bar\n", "Foo", false, "typedef Foo=bar\n")] +#[case("type Uuid = string\n", "Uuid", false, "type Uuid = string\n")] +#[case("type Foo=bar\n", "Foo", false, "type Foo=bar\n")] #[case( - "typedef Record = (name: string, active: bool)\n", + "type Record = (name: string, active: bool)\n", "Record", false, - "typedef Record = (name: string, active: bool)\n" + "type Record = (name: string, active: bool)\n" )] #[case("extern type Handle\n", "Handle", true, "extern type Handle\n")] #[case("extern type Extra \n", "Extra", true, "extern type Extra \n")] -fn typedef_variations( +fn type_definition_variations( #[case] src: &str, #[case] expected: &str, #[case] is_extern: bool, @@ -77,8 +77,8 @@ fn typedef_variations( assert_no_parse_errors(parsed.errors()); let defs = parsed.root().type_defs(); assert_eq!(defs.len(), 1); - #[expect(clippy::expect_used, reason = "tests require a typedef")] - let def = defs.first().expect("typedef should exist for valid source"); + #[expect(clippy::expect_used, reason = "tests require a type")] + let def = defs.first().expect("type should exist for valid source"); assert_eq!(def.name().as_deref(), Some(expected)); assert_eq!(def.is_extern(), is_extern); let text = pretty_print(def.syntax()); @@ -86,32 +86,38 @@ fn typedef_variations( } #[rstest] -fn typedef_nesting_and_whitespace() { - let src = "typedef Foo=string\ntypedef Bar = u64 \n"; +fn type_definition_nesting_and_whitespace() { + let src = "type Foo=string\ntype Bar = u64 \n"; let parsed = parse(src); assert_no_parse_errors(parsed.errors()); assert_eq!(pretty_print(parsed.root().syntax()), src); let defs = parsed.root().type_defs(); assert_eq!(defs.len(), 2); - #[expect(clippy::expect_used, reason = "tests require both typedefs to exist")] - let first = defs.first().expect("first typedef missing"); - #[expect(clippy::expect_used, reason = "tests require both typedefs to exist")] - let second = defs.get(1).expect("second typedef missing"); - assert_eq!(pretty_print(first.syntax()), "typedef Foo=string\n"); - assert_eq!(pretty_print(second.syntax()), "typedef Bar = u64 \n"); + #[expect( + clippy::expect_used, + reason = "tests require both type definitions to exist" + )] + let first = defs.first().expect("first type missing"); + #[expect( + clippy::expect_used, + reason = "tests require both type definitions to exist" + )] + let second = defs.get(1).expect("second type missing"); + assert_eq!(pretty_print(first.syntax()), "type Foo=string\n"); + assert_eq!(pretty_print(second.syntax()), "type Bar = u64 \n"); let first_end = first.syntax().text_range().end(); let second_start = second.syntax().text_range().start(); assert!(first_end <= second_start); } #[rstest] -fn typedef_missing_name_returns_none() { - let src = "typedef = string\n"; +fn type_definition_missing_name_returns_none() { + let src = "type = string\n"; let parsed = parse(src); assert_no_parse_errors(parsed.errors()); let defs = parsed.root().type_defs(); assert_eq!(defs.len(), 1); - #[expect(clippy::expect_used, reason = "tests require a typedef span")] - let def = defs.first().expect("typedef span exists"); + #[expect(clippy::expect_used, reason = "tests require a type span")] + let def = defs.first().expect("type span exists"); assert_eq!(def.name(), None); } diff --git a/src/parser/validators/name_uniqueness.rs b/src/parser/validators/name_uniqueness.rs index b4f026e3..ee269bc1 100644 --- a/src/parser/validators/name_uniqueness.rs +++ b/src/parser/validators/name_uniqueness.rs @@ -23,7 +23,7 @@ use crate::{Span, SyntaxKind}; /// use ddlint::parse; /// use ddlint::parser::validators::name_uniqueness::validate_name_uniqueness; /// -/// let parsed = parse("typedef A = u32\ntypedef A = string"); +/// let parsed = parse("type A = u32\ntype A = string"); /// let errors = validate_name_uniqueness(parsed.root()); /// assert_eq!(errors.len(), 1); /// ``` @@ -167,8 +167,8 @@ mod tests { #[test] fn no_duplicates_no_errors() { let src = concat!( - "typedef A = u32\n", - "typedef B = string\n", + "type A = u32\n", + "type B = string\n", "input relation R(x: u32)\n", "index I(x: u32) on R[x]\n", ); @@ -178,7 +178,7 @@ mod tests { } #[rstest] - #[case("typedef A = u32\ntypedef A = string", "duplicate type name 'A'")] + #[case("type A = u32\ntype A = string", "duplicate type name 'A'")] #[case( "input relation R(x: u32)\noutput relation R(y: string)\n", "duplicate relation name 'R'" @@ -213,8 +213,8 @@ mod tests { #[test] fn malformed_item_skipped() { - // A typedef without a name should not cause a false positive - let src = "typedef = u32\ntypedef A = string"; + // A type without a name should not cause a false positive + let src = "type = u32\ntype A = string"; let parsed = parse(src); let errors = super::validate_name_uniqueness(parsed.root()); assert!( diff --git a/src/sema/tests.rs b/src/sema/tests.rs index f8c307eb..1e923dc8 100644 --- a/src/sema/tests.rs +++ b/src/sema/tests.rs @@ -82,7 +82,7 @@ fn semantic_model(#[default("")] source: &str) -> super::SemanticModel { #[case("User", DeclarationKind::Relation)] fn collects_program_scope_declarations( #[with(concat!( - "typedef UserId = u32\n", + "type UserId = u32\n", "function project(id: UserId): UserId {}\n", "input relation User(id: UserId)" ))] diff --git a/src/sema/tests/name_span.rs b/src/sema/tests/name_span.rs index f0154ef4..c40cedb3 100644 --- a/src/sema/tests/name_span.rs +++ b/src/sema/tests/name_span.rs @@ -53,7 +53,7 @@ fn find_symbol<'a>( "project" )] #[case( - "typedef UserId = u64", + "type UserId = u64", "UserId", DeclarationKind::Type, SymbolOrigin::TypeDeclaration, diff --git a/src/test_util/assertions.rs b/src/test_util/assertions.rs index e473b7e1..41a33ac1 100644 --- a/src/test_util/assertions.rs +++ b/src/test_util/assertions.rs @@ -1,6 +1,6 @@ //! Assertion helpers for verifying parser errors in tests. -use super::{ErrorPattern, normalise_tokens}; +use super::{ErrorPattern, normalize_tokens}; use crate::SyntaxKind; use chumsky::error::{Simple, SimpleReason}; use std::ops::Range; @@ -121,9 +121,9 @@ pub fn assert_parse_error( panic!("error missing"); }; let rendered = format!("{error:?}"); - let rendered_normalised = normalise_tokens(&rendered); + let rendered_normalised = normalize_tokens(&rendered); let pattern_normalised = match &pattern { - ErrorPattern::Custom(msg) => normalise_tokens(msg), + ErrorPattern::Custom(msg) => normalize_tokens(msg), }; assert!( rendered_normalised.contains(&pattern_normalised), @@ -135,9 +135,9 @@ pub fn assert_parse_error( /// Return `true` if any error in `errors` is a [`SimpleReason::Custom`] whose /// normalised message contains the normalised `pattern`. fn any_custom_error_contains(errors: &[Simple], pattern: &str) -> bool { - let normalised = normalise_tokens(pattern); + let normalised = normalize_tokens(pattern); errors.iter().any(|error| match error.reason() { - SimpleReason::Custom(message) => normalise_tokens(message).contains(&normalised), + SimpleReason::Custom(message) => normalize_tokens(message).contains(&normalised), _ => false, }) } @@ -169,7 +169,7 @@ pub fn assert_custom_parse_error_contains( assert!( any_custom_error_contains(errors, pattern), "expected custom error containing `{}`, got {errors:?}", - normalise_tokens(pattern), + normalize_tokens(pattern), ); } @@ -199,7 +199,7 @@ pub fn assert_no_custom_parse_error_contains( assert!( !any_custom_error_contains(errors, pattern), "expected no custom error containing `{}`, but found one in {errors:?}", - normalise_tokens(pattern), + normalize_tokens(pattern), ); } @@ -226,11 +226,11 @@ pub fn assert_no_custom_parse_error_contains( #[must_use] pub fn find_matching_error(errors: &[Simple], pattern: &ErrorPattern) -> Option { let pattern_normalised = match pattern { - ErrorPattern::Custom(msg) => normalise_tokens(msg), + ErrorPattern::Custom(msg) => normalize_tokens(msg), }; errors.iter().position(|error| { let rendered = format!("{error:?}"); - let rendered_normalised = normalise_tokens(&rendered); + let rendered_normalised = normalize_tokens(&rendered); rendered_normalised.contains(&pattern_normalised) }) } @@ -245,9 +245,9 @@ fn assert_delimiter_error_impl<'a>( panic!("error missing"); }; let rendered = format!("{error:?}"); - let rendered_normalised = normalise_tokens(&rendered); + let rendered_normalised = normalize_tokens(&rendered); let pattern_normalised = match expected_pattern { - ErrorPattern::Custom(msg) => normalise_tokens(msg), + ErrorPattern::Custom(msg) => normalize_tokens(msg), }; assert!( rendered_normalised.contains(&pattern_normalised), diff --git a/src/test_util/mod.rs b/src/test_util/mod.rs index 26a056d3..7fd9e393 100644 --- a/src/test_util/mod.rs +++ b/src/test_util/mod.rs @@ -100,7 +100,7 @@ impl From<&str> for ErrorPattern { } /// Replace internal token names with human-readable forms. -pub(crate) fn normalise_tokens(s: &str) -> String { +pub(crate) fn normalize_tokens(s: &str) -> String { // Build replacements from SyntaxKind debug names to human-friendly labels. // This avoids a hand-maintained token map drifting from the parser. use SyntaxKind as K; diff --git a/tests/attribute_placement.rs b/tests/attribute_placement.rs index c070749b..595a1d40 100644 --- a/tests/attribute_placement.rs +++ b/tests/attribute_placement.rs @@ -9,7 +9,7 @@ use ddlint::test_util::assert_no_parse_errors; use rstest::rstest; #[rstest] -#[case("#[cold]\ntypedef T = u32")] +#[case("#[cold]\ntype T = u32")] #[case("#[inline]\nfunction f() {}")] #[case("#[hot]\ninput relation R(x: u32)")] #[case("#[hot]\noutput relation R(x: u32)")] @@ -18,13 +18,13 @@ use rstest::rstest; #[case("#[hot]\nmultiset R(x: u32)")] #[case("#[cold]\nextern function f()")] #[case("#[cold]\nextern type Handle")] -#[case("#[a]\n#[b]\ntypedef T = u32")] +#[case("#[a]\n#[b]\ntype T = u32")] fn valid_attribute_placement(#[case] src: &str) { let parsed = parse(src); assert_no_parse_errors(parsed.errors()); } -/// Spec §12: attributes are only permitted on typedef, function, and +/// Spec §12: attributes are only permitted on type, function, and /// relation declarations. #[rstest] #[case("#[cold]\nindex Ix(a: T) on A(a)")] diff --git a/tests/name_uniqueness.rs b/tests/name_uniqueness.rs index 960e1a66..2654aa8a 100644 --- a/tests/name_uniqueness.rs +++ b/tests/name_uniqueness.rs @@ -10,8 +10,8 @@ use rstest::rstest; #[test] fn mixed_program_no_duplicates() { let src = concat!( - "typedef A = u32\n", - "typedef B = string\n", + "type A = u32\n", + "type B = string\n", "input relation R(x: u32)\n", "output relation S(y: string)\n", "index IR(x: u32) on R[x]\n", @@ -27,7 +27,7 @@ fn mixed_program_no_duplicates() { } #[rstest] -#[case("typedef A = u32\ntypedef A = string", "duplicate", "A")] +#[case("type A = u32\ntype A = string", "duplicate", "A")] #[case( "input relation R(x: u32)\noutput relation R(y: string)\n", "duplicate", @@ -64,8 +64,8 @@ fn duplicate_detected(#[case] src: &str, #[case] expected_word: &str, #[case] ex #[test] fn multiple_duplicate_categories_all_reported() { let src = concat!( - "typedef A = u32\n", - "typedef A = string\n", + "type A = u32\n", + "type A = string\n", "index I(x: u32) on R[x]\n", "index I(y: string) on S[y]\n", "import foo\n", @@ -80,7 +80,7 @@ fn multiple_duplicate_categories_all_reported() { assert_eq!( dup_errors.len(), 3, - "expected three duplicate errors (typedef, index, import): {dup_errors:?}" + "expected three duplicate errors (type, index, import): {dup_errors:?}" ); } diff --git a/tests/reserved_token_rejection.rs b/tests/reserved_token_rejection.rs new file mode 100644 index 00000000..5042c265 --- /dev/null +++ b/tests/reserved_token_rejection.rs @@ -0,0 +1,133 @@ +//! Behavioural tests for reserved-token rejection. +//! +//! These cases exercise the public `parse()` entrypoint so parser-policy +//! diagnostics remain visible to downstream linter and semantic-model callers. + +use ddlint::linter::{CstRule, CstRuleStore, LintDiagnostic, Rule, RuleConfig, RuleCtx, Runner}; +use ddlint::test_util::{assert_custom_parse_error_contains, assert_no_parse_errors}; +use ddlint::{DdlogLanguage, SyntaxKind, SyntaxToken, parse}; +use rstest::rstest; + +const RESERVED_TYPEDEF_ERROR: &str = "`typedef` is a legacy DDlog keyword; use `type` instead"; +const RESERVED_SPACESHIP_ERROR: &str = + "`<=>` was reserved upstream but has no semantics in DDlog; remove it"; +const RESERVED_BARE_HASH_ERROR: &str = + "`#` is reserved; only `#[...]` attribute syntax is accepted"; +const RESERVED_BIGINT_ERROR: &str = + "`bigint` is a legacy type name; use a sized integer such as `i64` or `u64`"; +const RESERVED_BIT_ERROR: &str = + "`bit` is a legacy type name; use an unsigned sized integer such as `u32`"; +const RESERVED_DOUBLE_ERROR: &str = "`double` is a legacy type name; use `f64`"; +const RESERVED_FLOAT_ERROR: &str = "`float` is a legacy type name; use `f32`"; +const RESERVED_SIGNED_ERROR: &str = + "`signed` is a legacy type name; use a signed sized integer such as `i32`"; + +#[rstest] +#[case::typedef("typedef Foo = u32\n", RESERVED_TYPEDEF_ERROR)] +#[case::spaceship("Output(x) :- Source(x), x <=> 1.\n", RESERVED_SPACESHIP_ERROR)] +#[case::bare_hash("# foo\n", RESERVED_BARE_HASH_ERROR)] +#[case::bigint("type Foo = bigint;\n", RESERVED_BIGINT_ERROR)] +#[case::bit("type Foo = bit<32>;\n", RESERVED_BIT_ERROR)] +#[case::double("type Foo = double;\n", RESERVED_DOUBLE_ERROR)] +#[case::float("type Foo = float;\n", RESERVED_FLOAT_ERROR)] +#[case::signed("type Foo = signed<32>;\n", RESERVED_SIGNED_ERROR)] +fn reserved_tokens_emit_public_parse_errors(#[case] source: &str, #[case] expected: &str) { + let parsed = parse(source); + + assert_custom_parse_error_contains(parsed.errors(), expected); +} + +#[test] +fn import_alias_keyword_still_parses_cleanly() { + let parsed = parse("import foo::bar as baz\n"); + + assert_no_parse_errors(parsed.errors()); + let imports = parsed.root().imports(); + assert_eq!(imports.len(), 1); + assert_eq!( + imports.first().map(ddlint::ast::Import::path).as_deref(), + Some("foo::bar") + ); + assert_eq!( + imports + .first() + .and_then(ddlint::ast::Import::alias) + .as_deref(), + Some("baz") + ); +} + +#[rstest] +#[case("# cold\n")] +#[case("# [cold]\ntype Foo = u32\n")] +#[case("#/*comment*/[cold]\ntype Foo = u32\n")] +fn bare_hash_is_rejected(#[case] source: &str) { + let bare = parse(source); + assert_custom_parse_error_contains(bare.errors(), RESERVED_BARE_HASH_ERROR); +} + +#[test] +fn attribute_hash_is_preserved() { + let attributed = parse("#[cold]\ntype Foo = u32\n"); + assert_no_parse_errors(attributed.errors()); + assert_eq!( + attributed + .root() + .syntax() + .descendants() + .filter(|node| node.kind() == SyntaxKind::N_ATTRIBUTE) + .count(), + 1 + ); + assert_eq!(attributed.root().type_defs().len(), 1); +} + +#[test] +fn runner_skips_rules_for_any_parse_error() { + let source = "input relation R(x: u32); $"; + let parsed = parse(source); + + assert!( + !parsed.errors().is_empty(), + "generic syntax error should prevent lint rule execution" + ); + + let mut store = CstRuleStore::new(); + store.register(Box::new(ParseErrorSentinelRule)); + let diagnostics = Runner::new(&store, source, &parsed, RuleConfig::new()).run(); + assert!( + diagnostics.is_empty(), + "parse errors should prevent registered rules from running: {diagnostics:?}", + ); +} + +struct ParseErrorSentinelRule; + +impl Rule for ParseErrorSentinelRule { + fn name(&self) -> &'static str { + "parse-error-sentinel" + } + + fn group(&self) -> &'static str { + "test" + } + + fn docs(&self) -> &'static str { + "Panics if a parser error permits lint rule execution." + } +} + +impl CstRule for ParseErrorSentinelRule { + fn target_kinds(&self) -> &'static [SyntaxKind] { + &[SyntaxKind::K_RELATION] + } + + fn check_token( + &self, + _token: &SyntaxToken, + _ctx: &RuleCtx, + _diagnostics: &mut Vec, + ) { + panic!("parse errors must prevent lint rule execution"); + } +} diff --git a/typos.local.toml b/typos.local.toml index aadc3ed2..90fce0d5 100644 --- a/typos.local.toml +++ b/typos.local.toml @@ -6,8 +6,8 @@ schema = 1 stems = [] [words] -# Upstream titles and the established `Root::applys` parser API. -accepted = ["Center", "Flavored", "applys"] +# Upstream titles, external API spellings, and established parser APIs. +accepted = ["Center", "Flavored", "applys", "color"] [words.corrections] diff --git a/typos.toml b/typos.toml index 9a229a27..7fb02a1a 100644 --- a/typos.toml +++ b/typos.toml @@ -32,7 +32,6 @@ locale = "en-gb" extend-ignore-re = [ "(?s)```.*?```", "\\brust-analyzer\\b", - "`[^`\\n]+`", ] [default.extend-words] @@ -310,6 +309,7 @@ extend-ignore-re = [ "colonizers" = "colonizers" "colonizes" = "colonizes" "colonizing" = "colonizing" +"color" = "color" "colourisable" = "colourizable" "colourisation" = "colourization" "colourisations" = "colourizations"