Implement logos-based tokenizer - #7

Merged

leynos merged 8 commits into

mainfrom

codex/build-differential-datalog-tokenizer

Jun 25, 2025

Commits on Jun 24, 2025

Add tokenizer using logos
leynos
committed
21ac3ff
Add tokenizer tests
leynos
committed
cf35d9f
Add tokenizer mermaid diagram
leynos
committed
19f9ed1
Add module docs and expose tokenizer
leynos
committed
e2db157
Refine tokenizer tests and dependencies
leynos
committed
9625b8c
Improve tokenizer and tests
leynos
committed
23a5207

Commits on Jun 25, 2025

Add comprehensive tokenizer tests
leynos
committed
373ae27
Optimise keyword lookup and refine tests
leynos
committed
6844bf6