Support Parallel Data Loading Shufflable Iterable Datasets/DataStreams - #100

Open

alex-jw-brooks wants to merge 6 commits into

caikit:mainfrom

alex-jw-brooks:parallel_streams

Commits on Jul 25, 2023

Preliminary implementation for distributed shufflable streams
alex-jw-brooks
committed
376cef2
Add stream partition docstrings and type hints
alex-jw-brooks
committed
67ee5a3
Add test for multiworker shuffling
alex-jw-brooks
committed
89b2858
Stream wrapper linter and code formatting
alex-jw-brooks
committed
b7bd4cb
Add notice for multiworker shuffling behaviors
alex-jw-brooks
committed
b6d1b82
Cache datastream length
alex-jw-brooks
committed
1392991