Skip to content

Add lexer fast path for o200k_base, cl100k_base, and gpt-2 pretokenization patterns; custom regexes fall back to fancy_regex#552

Open
augustasio wants to merge 3 commits into
openai:mainfrom
augustasio:lexer-pretokenization
Open

Add lexer fast path for o200k_base, cl100k_base, and gpt-2 pretokenization patterns; custom regexes fall back to fancy_regex#552
augustasio wants to merge 3 commits into
openai:mainfrom
augustasio:lexer-pretokenization

Lexer: propagate fancy_regex errors in encode (regex fallback path)

4cf7a9d
Select commit
Loading
Failed to load commit list.

Workflow runs completed with no jobs