Causal LM tokenization: Chunking and seq2seq Forward - #206

Merged

alex-jw-brooks merged 23 commits into

caikit:mainfrom

alex-jw-brooks:causal_lm_tok_toggle

Oct 2, 2023

Commits on Sep 29, 2023

Add forward for seq2seq tokenization
alex-jw-brooks
committed
7e767b2
Add comparator test for seq2seq forwarding
alex-jw-brooks
committed
4c899dc
Split seq2seq tokenization preprocessing
alex-jw-brooks
committed
a159209
Add forward to seq2seq tokenization (no batch)
alex-jw-brooks
committed
ad323a9
Add batch forwarding tests for seq2seq/causal lm
alex-jw-brooks
committed
0d894ac
Add batch forward for causal lm / seq2seq
alex-jw-brooks
committed
fe2c57c
rewrite causal lm tok tests to check chunking
alex-jw-brooks
committed
727e784
Implement chunked tokenization for causal lm
alex-jw-brooks
committed
ac9fedc
linting, formatting
alex-jw-brooks
committed
5fc9849
Turn on seq2seq tokenization by default
alex-jw-brooks
committed
c514d68
Hack: explictly force stream unwrapping to false
alex-jw-brooks
committed
0b75f80
Approximate port of old causal lm tokenization logic
alex-jw-brooks
committed
52a33a5
Hack - use default collator for causal LM
alex-jw-brooks
committed
9f1aff3
Add generic test for left/right padding causal lm seq approach
alex-jw-brooks
committed
33e3e14
Do left / right padding via tokenizer pad
alex-jw-brooks
committed
a515e35
Add simple tests for default data collator
alex-jw-brooks
committed
73e804e
Update concat seq test for corrected padding
alex-jw-brooks
committed
349cbdc
Fix legacy ported sequence length bug
alex-jw-brooks
committed
ed1f8e6
Update comments for tokenizer changes
alex-jw-brooks
committed
579db79
linting and formatting
alex-jw-brooks
committed
560d5bc
Update causal lm docstrings and type hints
alex-jw-brooks
committed
511e561
Fix remainder handling in chunking test
alex-jw-brooks
committed
ccd9c13

Commits on Oct 2, 2023

Add chunk example, use extend
alex-jw-brooks
committed
52f9910