# Common Voice 8.0.0 en using all available RAM

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcommon-voice-8-0-0-en-using-all-available-ram%2F16541%3E%5C
**Category:** 🤗Datasets\
**Created:** [April 6, 2022, 3:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcommon-voice-8-0-0-en-using-all-available-ram%2F16541 "2022-04-06T15:00:40Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ollie](https://avatars.discourse-cdn.com/v4/letter/o/a87d85/32.png) [@Ollie](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOllie)%5C
**Post date:** [April 6, 2022, 3:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcommon-voice-8-0-0-en-using-all-available-ram%2F16541%2F1 "2022-04-06T15:00:40Z")

</div>

Hello. In the past I’ve been able to download prior Common Voice releases using `common_voice` and the version number, but `mozilla-foundation/common_voice_8_0` seems to give me memory issues.

The code snippet below downloads the dataset just fine but uses up all available memory in a Colab High-RAM environment when preparing the train set:

```python
dataset = datasets.load_dataset("mozilla-foundation/common_voice_8_0", 
                                "en",
                                use_auth_token="my_auth_token",
                                split="train")

```

It’s the English dataset so this will take time to download. I reduced the `writer_batch_size` in the hopes that it will work but no luck ☹

Any help would be greatly appreciated!

---

<div class="post-metadata">

**Author:** ![Ollie](https://avatars.discourse-cdn.com/v4/letter/o/a87d85/32.png) [@Ollie](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOllie)%5C
**Post date:** [April 7, 2022, 6:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcommon-voice-8-0-0-en-using-all-available-ram%2F16541%2F2 "2022-04-07T06:50:17Z")

</div>

Pinging @lhoestq because this could be related to [this issue](https://github.com/huggingface/datasets/issues/4057) maybe?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [April 7, 2022, 9:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcommon-voice-8-0-0-en-using-all-available-ram%2F16541%2F3 "2022-04-07T09:19:55Z")

</div>

Hi ! On which version of `datasets` do you have this issue ? and which version of `pyarrow` ?

---

<div class="post-metadata">

**Author:** ![Ollie](https://avatars.discourse-cdn.com/v4/letter/o/a87d85/32.png) [@Ollie](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOllie)%5C
**Post date:** [April 7, 2022, 10:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcommon-voice-8-0-0-en-using-all-available-ram%2F16541%2F4 "2022-04-07T10:00:50Z")

</div>

Sorry, should have provided that first.  
`datasets` was installed from `master` so the version is `2.0.1.dev0`, the `pyarrow` version is `6.0.1`.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [April 7, 2022, 10:22am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcommon-voice-8-0-0-en-using-all-available-ram%2F16541%2F5 "2022-04-07T10:22:47Z")

</div>

Thanks ! I was able to reproduce with datasets 2.0.0 and pyarrow 6.0.1 on colab, I’ll investigate

---

<div class="post-metadata">

**Author:** ![Ollie](https://avatars.discourse-cdn.com/v4/letter/o/a87d85/32.png) [@Ollie](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOllie)%5C
**Post date:** [April 19, 2022, 9:44am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcommon-voice-8-0-0-en-using-all-available-ram%2F16541%2F6 "2022-04-19T09:44:58Z")

</div>

Hi @lhoestq, don’t mean to be a pain but were you able to find the cause of this? Thanks!

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [April 19, 2022, 9:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcommon-voice-8-0-0-en-using-all-available-ram%2F16541%2F7 "2022-04-19T09:56:49Z")

</div>

Hi, not yet but investigating this is indeed in my very short term todo list 😉 Will keep you posted !

---

<div class="post-metadata">

**Author:** ![Ollie](https://avatars.discourse-cdn.com/v4/letter/o/a87d85/32.png) [@Ollie](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOllie)%5C
**Post date:** [August 5, 2022, 1:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcommon-voice-8-0-0-en-using-all-available-ram%2F16541%2F8 "2022-08-05T13:18:53Z")

</div>

[Fixed](https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0/discussions/2)
