Add lexer fast path for o200k_base, cl100k_base, and gpt-2 pretokenization patterns; custom regexes fall back to fancy_regex#552

Open

augustasio wants to merge 3 commits into

openai:mainfrom

augustasio:lexer-pretokenization

Commits on May 22, 2026

Add lexer fast path for o200k_base, cl100k_base, and gpt-2 pretokenization patterns; custom regexes fall back to fancy_regex
augustasm
committed
Lexer: name the two magic numbers (ASCII_BOUNDARY, MAX_DIGIT_RUN)
augustasm
committed
Lexer: propagate fancy_regex errors in encode (regex fallback path)
augustasm
committed