# Improve performance IterableDataset (with tokenization)
**URL:**
**Author:**  [@cwallenwein](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcwallenwein)%5C
**Post date:** [October 27, 2023, 9:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fimprove-performance-iterabledataset-with-tokenization%2F60141%2F1 "2023-10-27T09:00:52Z")
Hi friends 👋
**my problem**
I’m using an IterableDataset to load a big dataset and I’m tokenizing it on demand using the IterableDatase.map function. For most samples, the loading time is reasonable, but sometimes, retrieving the next sample takes a really long time (45 sec). This typically happens for very long samples. Any ideas of how I can reduce the waiting time?
**possible solutions**
I’m currently testing if it’s possible to preload samples from the dataset using the iterlib package. Any other suggestions?
**my tokenization code**
(taken from the [Data processing for Causal Language Modeling](https://www.youtube.com/watch?v=ma1TrR7gE7I) video by Huggingface)
```auto
def tokenize(batch, tokenizer, context_length):
outputs = tokenizer(
batch["text"],
truncation=True,
max_length=context_length,
return_overflowing_tokens=True,
return_length=True
)
input_batch = []
for length, input_ids in zip(outputs["length"], outputs["input_ids"]):
if length == context_length:
input_batch.append(input_ids)
return {"input_ids": input_batch}
```
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [October 29, 2023, 3:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fimprove-performance-iterabledataset-with-tokenization%2F60141%2F2 "2023-10-29T15:28:25Z")
Maybe you can add another `map` operation before tokenization to split long sentences ?
---
**Author:**  [@cwallenwein](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcwallenwein)%5C
**Post date:** [November 2, 2023, 9:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fimprove-performance-iterabledataset-with-tokenization%2F60141%2F3 "2023-11-02T09:48:06Z")
Thanks for the suggestion. I tried that but it still doesn’t really work. Maybe the problem is also loading the data from memory instead of tokenizing it 🤔