Add parser diagnostic observability infrastructure (#304)

Cargo.toml

@@ -16,8 +16,11 @@ thiserror = { version = "1.0", default-features = false }
 derive_more = { version = "2.1", default-features = false, features = ["from", "as_ref"] }
 rayon = "1.10"
 smallvec = "1.15.1"
+metrics = { version = "0.24.6", default-features = false, optional = true }
+tracing = { version = "0.1.44", default-features = false, optional = true }
 
 [features]
+observability = ["dep:metrics", "dep:tracing"]
 test-support = []
 
 [dev-dependencies]

docs/contents.md

@@ -30,6 +30,8 @@
 - [Parser conformance register](./parser-conformance-register.md): Tracker for
   specification and implementation deltas, resolved contradictions, and open
   parser design decisions.
+- [Parser observability](./parser-observability.md): How to export parser
+  attempt and diagnostic events to downstream logging and metrics backends.
 - [Haskell parser analysis](./haskell-parser-analysis.md): Reference analysis
   of the upstream DDlog parser behaviour used when validating parser
   compatibility.

docs/parser-conformance-register.md

@@ -193,6 +193,21 @@ This register tracks parser behaviour against the syntax specification.
 - Decision status: `scheduled`.
 - Roadmap item: `docs/roadmap.md` item `2.6.8`.
 
+## Stable diagnostic compatibility surface
+
+Issue `#304` establishes a backend-neutral observability precursor to ADR-001
+Phase 2. `DiagnosticCode` freezes `D-REL-001` through `D-REL-008` as stable
+identifiers, while `DiagnosticCategory` classifies every current scanner and
+parser orchestration merge point. Existing code meanings do not change; adding
+a code is additive. Consumers must not treat human-facing message text as an
+identifier.
+
+`parse_with_observer()` reports attempt and diagnostic records without changing
+the `Parsed` result. `parse()` uses the no-op observer and remains independent
+of any logging or metrics runtime. This does not complete roadmap item `2.8.1`,
+which still requires the post-split public diagnostic contract and stage
+modelling.
+
 ## Maintenance rules
 
 When parser behaviour changes, update this register in the same change:

docs/parser-implementation-notes.md

@@ -50,6 +50,9 @@ Current pipeline guarantees are intentionally narrow:
 
 - `parse()` builds the CST-backed `Parsed` result, collects top-level `for`
   semantic rules, and runs top-level name-uniqueness validation.
+- `parse_with_observer()` reports the same pipeline through a backend-neutral
+  observer. Observer callbacks do not return parser-control decisions, and
+  `parse()` supplies the no-op observer.
 - `parse()` does **not** classify rule-body aggregations or report duplicate or
   wrong-arity aggregation diagnostics in `Parsed::errors()`.
 - Aggregation classification and validation happen when callers request
@@ -379,6 +382,9 @@ token names or their human-readable equivalents.
 
 - Tokenization and keyword policy: `src/tokenizer.rs`
 - Entry parse orchestration: `src/parser/mod.rs`
+- Stable diagnostic taxonomy: `src/parser/diagnostics/*`
+- Backend-neutral observer and optional adapter:
+  `src/parser/observability/*`
 - Pratt parser: `src/parser/expression/pratt.rs`
 - Pratt postfix helpers:
   `src/parser/expression/pratt/{postfix,diff,delay}.rs`

docs/parser-observability.md

@@ -0,0 +1,144 @@
+# How to export parser observability
+
+This guide shows host-application maintainers how to collect parser attempts
+and diagnostics without coupling parser behaviour to a telemetry backend. Use
+it when a service, command-line application, or batch worker needs parser logs,
+metrics, or alerts.
+
+## Prerequisites
+
+- A host application that calls `ddlint::parse`.
+- An existing logging or metrics backend if events must leave the process.
+- The `observability` feature only when using the supplied `tracing` and
+  `metrics` adapter. Custom observers do not need this feature.
+
+The library never installs a tracing subscriber, metrics recorder, or exporter.
+Only the host application initializes global telemetry.
+
+## Choose an observer
+
+For a custom backend, implement `ParseObserver` and copy any borrowed
+diagnostic data needed after the callback:
+
+```rust
+use ddlint::{
+    DiagnosticContext, ParseAttemptContext, ParseObserver, parse_with_observer,
+};
+
+struct HostObserver;
+
+impl ParseObserver for HostObserver {
+    fn parse_attempt_completed(&self, context: ParseAttemptContext) {
+        host_metrics::record_attempt(
+            context.category().as_str(),
+            context.diagnostic_count(),
+        );
+    }
+
+    fn diagnostic_emitted(&self, context: &DiagnosticContext<'_>) {
+        host_log::record_parser_failure(
+            context.code().map(|code| code.as_str()),
+            context.category().as_str(),
+            context.span(),
+            context.severity().as_str(),
+            context.message(),
+        );
+    }
+}
+
+let parsed = parse_with_observer(source, &HostObserver);
+```
+
+Do not use message text, source spans, source text, paths, or request
+identifiers as metrics labels. They are unbounded and belong in structured logs
+or traces.
+
+For the supplied facade adapter, enable the feature:
+
+```toml
+[dependencies]
+ddlint = { version = "0.1", features = ["observability"] }
+```
+
+Initialize the application's existing tracing subscriber and metrics recorder,
+then pass `TelemetryObserver`:
+
+```rust
+use ddlint::{TelemetryObserver, parse_with_observer};
+
+// The application initializes its subscriber, recorder, and exporters first.
+let observer = TelemetryObserver::new();
+let parsed = parse_with_observer(source, &observer);
+```
+
+Without a configured subscriber or recorder, the facades discard events and
+parsing remains deterministic. Calling `parse(source)` uses `NoopParseObserver`
+and does not require a telemetry runtime.
+
+## Export logs and metrics
+
+`TelemetryObserver` emits the following counters:
+
+| Metric                            | Labels                         | Meaning                            |
+| --------------------------------- | ------------------------------ | ---------------------------------- |
+| `ddlint_parser_attempts_total`    | `category`                     | Parser or scanner attempts started |
+| `ddlint_parser_diagnostics_total` | `code`, `category`, `severity` | Diagnostics emitted                |
+
+The `code` label is the stable diagnostic code or `uncoded`. Categories and
+severity are bounded enums. The adapter emits attempt start/completion events at
+`DEBUG` and diagnostics at `ERROR` under the `ddlint::parser` tracing target.
+Diagnostic events include `code`, `category`, `severity`, `span_start`,
+`span_end`, and `message` fields.
+
+Configure the host's normal OpenTelemetry, Prometheus, or vendor exporter to
+collect these facade events. Keep exporter lifecycle, batching, retries, and
+shutdown in the application boundary.
+
+## Configure alerts
+
+Alert on rates and ratios over a sustained window rather than raw counter
+values:
+
+1. Use `ddlint_parser_attempts_total{category="parser"}` as the parse-volume
+   denominator.
+2. Alert when the diagnostic-to-parser-attempt ratio exceeds the
+   application's error budget after a minimum traffic threshold.
+3. Group failures by `category` to distinguish scanner regressions from lexer,
+   span-builder, top-level-`for`, or name-validation failures.
+4. Add focused alerts for unexpected increases in `D-REL-001` through
+   `D-REL-008` when relation input quality is operationally significant.
+5. Track `code="uncoded"` separately. A sustained increase can identify a
+   scanner family that needs its own stable code.
+
+Do not page solely because a diagnostic exists: invalid user input can be an
+expected outcome. Choose thresholds from the host application's baseline and
+route alerts to the team that owns its input pipeline.
+
+## Stable diagnostic contract
+
+The diagnostic-code list is a compatibility surface under ADR-001 Phase 2.
+Existing code meanings do not change; new codes may be added. Human-facing
+message wording may become clearer, so automation must match `DiagnosticCode` or
+`DiagnosticCategory`, not message text.
+
+| Code        | Relation failure                               |
+| ----------- | ---------------------------------------------- |
+| `D-REL-001` | Kind keyword appears before the role keyword   |
+| `D-REL-002` | More than one role keyword                     |
+| `D-REL-003` | More than one kind keyword                     |
+| `D-REL-004` | Bracket-form relation declares a primary key   |
+| `D-REL-005` | Bracket form lacks one element type            |
+| `D-REL-006` | Non-input relation declares a primary key      |
+| `D-REL-007` | Malformed primary-key clause                   |
+| `D-REL-008` | Unsupported bracket-wrapped primary-key clause |
+
+Uncoded diagnostics retain a stable category. Current categories are `parser`,
+`attribute`, `import`, `typedef`, `relation`, `index`, `function`,
+`transformer`, `apply`, `rule`, `lexer`, `span_builder`, `top_level_for`, and
+`name_uniqueness`.
+
+## See also
+
+- [Parser implementation notes](./parser-implementation-notes.md)
+- [Parser conformance register](./parser-conformance-register.md)
+- [ADR-001: parser crate split](./adr-001-parser-crate-split.md)

src/lib.rs

@@ -17,7 +17,17 @@ pub mod tokenizer;
 pub mod test_util;
 
 pub use language::{DdlogLanguage, SyntaxKind};
-pub use parser::{Parsed, ast, parse};
+#[cfg(feature = "observability")]
+pub use parser::observability::TelemetryObserver;
+pub use parser::{
+    Parsed, ast,
+    diagnostics::{DiagnosticCategory, DiagnosticCode},
+    observability::{
+        DiagnosticContext, DiagnosticSeverity, NoopParseObserver, ParseAttemptContext,
+        ParseObserver,
+    },
+    parse, parse_with_observer,
+};
 /// Re-exported for macro-generated rule handlers and downstream CST consumers
 /// so callers can use `ddlint`'s public syntax types without a direct `rowan`
 /// dependency.

src/parser/cst_builder/tree.rs

@@ -111,7 +111,7 @@ fn validate_token_span(span: &Span, src_len: usize) -> bool {
 ///
 /// let src = "import foo::bar;";
 /// let tokens = tokenize(src);
-/// let (spans, errors) = parse_tokens(&tokens, src);
+/// let (spans, errors) = parse_tokens(&tokens, src, &NoopParseObserver);
 /// assert!(errors.is_empty());
 /// let green = build_green_tree(&tokens, src, &spans);
 /// let root = Root::from_green(green);
@@ -183,7 +183,11 @@ mod tests {
     fn build_green_tree_round_trip() {
         let src = "import foo::bar;";
         let tokens = tokenize(src);
-        let (spans, errors) = parse_tokens(&tokens, src);
+        let (spans, errors) = parse_tokens(
+            &tokens,
+            src,
+            &crate::parser::observability::NoopParseObserver,
+        );
         assert!(errors.is_empty());
         let green = build_green_tree(&tokens, src, &spans);
         let root = crate::parser::ast::Root::from_green(green);
@@ -209,7 +213,11 @@ mod tests {
     fn build_green_tree_skips_oob_token_span_in_release() {
         let src = "import foo::bar;";
         let mut tokens = tokenize(src);
-        let (spans, errors) = parse_tokens(&tokens, src);
+        let (spans, errors) = parse_tokens(
+            &tokens,
+            src,
+            &crate::parser::observability::NoopParseObserver,
+        );
         assert!(errors.is_empty());
 
         tokens.push((SyntaxKind::K_IMPORT, src.len() + 1..src.len() + 2));
@@ -244,7 +252,11 @@ mod tests {
             "User(id, name) :- name == \"a\", id > 0.\n"
         );
         let tokens = tokenize(src);
-        let (spans, errors) = parse_tokens(&tokens, src);
+        let (spans, errors) = parse_tokens(
+            &tokens,
+            src,
+            &crate::parser::observability::NoopParseObserver,
+        );
         assert!(errors.is_empty());
         let green = build_green_tree(&tokens, src, &spans);
         let root = Root::from_green(green);