# \[SOLVED\] Dataset.map() is frozen on ELI5

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsolved-dataset-map-is-frozen-on-eli5%2F656%3E%5C
**Category:** 🤗Datasets\
**Created:** [August 10, 2020, 9:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsolved-dataset-map-is-frozen-on-eli5%2F656 "2020-08-10T09:17:33Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jung](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jung/32/21989_2.png) [@Jung](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJung)%5C
**Post date:** [August 10, 2020, 9:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsolved-dataset-map-is-frozen-on-eli5%2F656%2F1 "2020-08-10T09:17:33Z")

</div>

I have tried to prepare ELI5 to train with T5, based on [this wonderful notebook of Suraj Patil](https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb)

However, when I run `dataset.map()` on ELI5 to prepare `input_text, target_text`, `dataset.map` is **frozen** in the first hundreds examples. On the contrary, this works totally fine on SQUAD (80,000 examples). Both `nlp` version 0.3.0 and 0.4.0 cause frozen process . Also try various `pyarrow` versions from 0.16.0 / 0.17.0 / 1.0.0 also have the same frozen process.

Reproducible code can be found on [this colab notebook](https://colab.research.google.com/drive/14wttOTv3ky74B_c0kv5WrbgQjCF2fYQk?usp=sharing), where I also show that the same mapping function works fine on SQUAD, so the problem is likely due to ELI5 somehow.

* * *

**More Info :** instead of `map`, if I run `for` loop and apply function by myself, there’s no error and can finish within 10 seconds. However, `nlp dataset` is immutable , so I could not create a new key-value within the dataset directly ) .

I also notice that SQUAD texts are quite clean while ELI5 texts contain many special characters, not sure if this is the cause ?

---

<div class="post-metadata">

**Author:** ![Jung](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jung/32/21989_2.png) [@Jung](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJung)%5C
**Post date:** [August 11, 2020, 11:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsolved-dataset-map-is-frozen-on-eli5%2F656%2F2 "2020-08-11T23:57:18Z")

</div>

Fixed by amazing Quentin here:  
[https://github.com/huggingface/nlp/issues/482](https://github.com/huggingface/nlp/issues/482)

Thanks very much again!

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [August 24, 2020, 7:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsolved-dataset-map-is-frozen-on-eli5%2F656%2F3 "2020-08-24T19:50:49Z")

</div>

You’re welcome 🙂
