# Map with tokenize function stuck in the beginning

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmap-with-tokenize-function-stuck-in-the-beginning%2F133047%3E%5C
**Category:** 🤗Datasets\
**Created:** [December 26, 2024, 2:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmap-with-tokenize-function-stuck-in-the-beginning%2F133047 "2024-12-26T02:21:00Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [December 27, 2024, 6:41am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmap-with-tokenize-function-stuck-in-the-beginning%2F133047%2F2 "2024-12-27T06:41:27Z")

</div>

Maybe try IterableDataset?

> [@Training a Tokenizer on a Streamed Dataset](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-tokenizer-on-a-streamed-dataset%2F8026)%3A
>
> Hi, I’m trying to train a tokenizer on a dataset that uses streaming. I followed the instructions provided [here](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling#train-tokenizer), with the addition of streaming=True during the dataset loading step. However, it quickly failed as the IterableDataset class does not have a length property (unlike the normal Dataset class). How can I work around this issue without having to download the dataset file entirely, i.e. purely relying on dataset streaming? Here’s the [Colab link](https://colab.research.google.com/drive/1rOdAzmfdp2S1tGDk4iaiM2d4pai4QGES?usp=sharing) to reproduce the error. Any help would be …

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmap-with-tokenize-function-stuck-in-the-beginning%2F133047)._