# Unable to Train for a Long Time

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-train-for-a-long-time%2F31815%3E%5C
**Category:** 🤗Datasets\
**Created:** [February 16, 2023, 2:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-train-for-a-long-time%2F31815 "2023-02-16T02:05:31Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![zpn](https://avatars.discourse-cdn.com/v4/letter/z/bc8723/32.png) [@zpn](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzpn)%5C
**Post date:** [February 16, 2023, 2:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-train-for-a-long-time%2F31815%2F1 "2023-02-16T02:05:31Z")

</div>

Hi all,

I really love the datasets package! One issue that I’ve frequently run into recently is when using the a dataset in streaming mode. One case, I get throttled, similar to the error message below

```auto
huggingface_hub.utils._errors.HfHubHTTPError: 429 Client Error: Too Many Requests for url: https://huggingface.co/api/datasets/zpn/uniref50

```

which seems odd since I’m only using 1 node of 8 GPUs.

The other case is after about 10 or 15 hours, it seems that I get disconnected and then the file is incorrectly downloaded, similar to

```auto
Got disconnected from remote data host. Retrying in 5sec [2/20]
Got disconnected from remote data host. Retrying in 5sec [3/20]
Got disconnected from remote data host. Retrying in 5sec [4/20]
Got disconnected from remote data host. Retrying in 5sec [1/20]
Got disconnected from remote data host. Retrying in 5sec [2/20]
Failed to read file 'zstd://shard_00000.jsonl::https://huggingface.co/datasets/gonzalobenegas/mammalian-genomes-cds/resolve/4cc9cfe7b5377c3dda6040345a9a6a5546e7e162/data/validation/shard_00000.jsonl.zst' with error <class 'pyarrow.lib.ArrowInvalid'>: JSON parse error: Column() changed from object to string in row 0
1545it [13:44, 1.87it/s]

```

This is after many cycles through of this particular dataset so it seems like the file is getting partially downloaded and then read (which raises the error).

Are there any workarounds to this? Training these models is costly especially with having to restart.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [February 16, 2023, 12:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-train-for-a-long-time%2F31815%2F2 "2023-02-16T12:22:38Z")

</div>

Hi ! I opened a PR a while ago to ignore bad chunks of JSON data if there is an issue while streaming but never got it to merge: [Add error\_bad\_chunk to the JSON loader by lhoestq · Pull Request #2838 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/pull/2838). Let me know if something like this would help

Regarding rate limiting, please make sure you don’t have another shell or someone on the same network as you spamming Hugging Face ^^’

---

<div class="post-metadata">

**Author:** ![zpn](https://avatars.discourse-cdn.com/v4/letter/z/bc8723/32.png) [@zpn](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzpn)%5C
**Post date:** [February 16, 2023, 3:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-train-for-a-long-time%2F31815%2F3 "2023-02-16T15:05:22Z")

</div>

Hm that’s odd that I face the rate limiting from a 1 node scenario. Is there a hard limit to the number of requests per second?

And yeah something like this would definitely be helpful. Is it possible to retry/remove a download in the case it’s corrupted?

If the jobs keep failing, I’m going to unfortunately have to move off of datasets and reimplement everything to read from S3 or something like that

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [February 16, 2023, 5:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-train-for-a-long-time%2F31815%2F4 "2023-02-16T17:41:33Z")

</div>

> Is there a hard limit to the number of requests per second?

We faced some spam from bots recently, so we had to hardcode a limit (at least for free users).

> And yeah something like this would definitely be helpful. Is it possible to retry/remove a download in the case it’s corrupted?

In the PR I linked it skips the corrupted batch, but we can probably extend it to allow to retry instead by seeking to the latest read location.

---

<div class="post-metadata">

**Author:** ![zpn](https://avatars.discourse-cdn.com/v4/letter/z/bc8723/32.png) [@zpn](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzpn)%5C
**Post date:** [February 16, 2023, 11:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-train-for-a-long-time%2F31815%2F5 "2023-02-16T23:59:34Z")

</div>

That would also be great!