# Training a Tokenizer on a Streamed Dataset
**URL:**
**Author:**  [@w11wo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fw11wo)%5C
**Post date:** [July 5, 2021, 7:02am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-tokenizer-on-a-streamed-dataset%2F8026%2F1 "2021-07-05T07:02:30Z")
Hi, I’m trying to train a tokenizer on a dataset that uses streaming. I followed the instructions provided [here](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling#train-tokenizer), with the addition of `streaming=True` during the dataset loading step. However, it quickly failed as the `IterableDataset` class does not have a length property (unlike the normal `Dataset` class).
How can I work around this issue without having to download the dataset file entirely, i.e. purely relying on dataset streaming? Here’s the [Colab link](https://colab.research.google.com/drive/1rOdAzmfdp2S1tGDk4iaiM2d4pai4QGES?usp=sharing) to reproduce the error.
Any help would be greatly appreciated. Thanks!
Note: This [issue](https://github.com/huggingface/datasets/issues/2583) seems to be related to mine.
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [July 6, 2021, 12:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-tokenizer-on-a-streamed-dataset%2F8026%2F2 "2021-07-06T12:51:29Z")
Hi !
You can follow the instructions [here](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling#train-tokenizer) and use this batch iterator instead:
```python
def batch_iterator(batch_size=1000):
batch = []
for example in dataset:
batch.append(example["text"])
if len(batch) == batch_size:
yield batch
batch = []
if batch: # yield last batch
yield batch
```
---
**Author:**  [@w11wo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fw11wo)%5C
**Post date:** [July 6, 2021, 1:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-tokenizer-on-a-streamed-dataset%2F8026%2F3 "2021-07-06T13:04:01Z")
Thanks for the reply and solution @lhoestq.
Appreciate the effort you and your team are doing!
Cheers!
---
**Author:**  [@dotan1111](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdotan1111)%5C
**Post date:** [January 12, 2023, 2:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-tokenizer-on-a-streamed-dataset%2F8026%2F4 "2023-01-12T14:00:37Z")
I’m having problems to train the tokenizers on a very large dataset. I think the problem is in the “yield” done in the interator. Any change I could train the tokenizer without the yield?
---
**Author:**  [@rootacess](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frootacess)%5C
**Post date:** [May 30, 2023, 3:12am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-tokenizer-on-a-streamed-dataset%2F8026%2F5 "2023-05-30T03:12:00Z")
@w11wo You can do this to get the ` __len__ ` property:
```python
dataset.with_format("torch")
```
---
**Author:**  [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [January 31, 2024, 6:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-tokenizer-on-a-streamed-dataset%2F8026%2F6 "2024-01-31T06:52:51Z")
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.