# AutoTrain Advanced Error on Massive Dataset

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fautotrain-advanced-error-on-massive-dataset%2F161395%3E%5C
**Category:** 🤗AutoTrain\
**Created:** [July 1, 2025, 10:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fautotrain-advanced-error-on-massive-dataset%2F161395 "2025-07-01T22:47:39Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [July 2, 2025, 7:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fautotrain-advanced-error-on-massive-dataset%2F161395%2F2 "2025-07-02T07:50:21Z")

</div>

It would be easier if Streaming (`IterableDataset`) could be used, but I don’t know how to enable Streaming in AutoTrainAdvanced…

It might be quicker to split the dataset into several smaller datasets in advance.😅

> [@Best practices for a large dataset](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-practices-for-a-large-dataset%2F137632)%3A
>
> Hi, I have a ~1TB large dataset stored at the HF hub. I can download this on my disk and read it successfully. Nevertheless, it’s large enough that I can’t fit it in my RAM. What is the best practice for training a model on such a dataset? I tried loading the dataset with load\_dataset(..., streaming=True) and then having two buffers: one that is being loaded by training process onto GPU and one that, in a separate thread, streams from the dataset and fills up the other buffer. Then, when the i…

> **[LLM Finetuning with AutoTrain Advanced](https://huggingface.co/docs/autotrain/main/en/tasks/llm_finetuning)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

> <https://github.com/huggingface/autotrain-advanced/issues/282>
>
> Some datasets require extra arguments to be passed such \`path\` and \`name\` both b…eing present, or \`revision\` etc. Streaming can also be much for efficient for some datasets. It would be great if we could pass some of these as command line arguments, i.e. : \`autotrain llm --data-path=some-data-set --data-kwargs "streaming=true,revision=v1.2" ...\`.

> [@Load Dataset and Save as Parquet](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-dataset-and-save-as-parquet%2F94003)%3A
>
> I am curious whether we can directly save a dataset as parquet as in the original dataset at HF. For example [fineweb-edu/sample/10BT](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu/tree/main/sample/10BT) has 13 parquet files # download the dataset fw = load\_dataset( "HuggingFaceFW/fineweb-edu", name="sample-10BT, split="train", num\_proc=10, cache\_dir=cache\_dir, ) fw.to\_parquet(parquet\_dir) Loads these files as arrow although they are originally parquet, and when saving with to\_parquet() it converts arrow to a single parquet file. So my que…

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fautotrain-advanced-error-on-massive-dataset%2F161395)._