You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
To integrate fairseq and infinibatch, I need to get the size of the entire dataset. I used to chunked_dataset_iterator to read the text dataset. As a workaround, right now I'm reading the length from a custom config file with hardcorded values.
Is there a way to get the length of entire dataset while continuing the same functionality as current iterator?
The text was updated successfully, but these errors were encountered:
To integrate
fairseq
andinfinibatch
, I need to get the size of the entire dataset. I used tochunked_dataset_iterator
to read the text dataset. As a workaround, right now I'm reading the length from a custom config file with hardcorded values.Is there a way to get the length of entire dataset while continuing the same functionality as current iterator?
The text was updated successfully, but these errors were encountered: