# AutoTrain Advanced Error on Massive Dataset
**URL:**
**Author:**  [@farranmedia](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ffarranmedia)%5C
**Post date:** [July 1, 2025, 10:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fautotrain-advanced-error-on-massive-dataset%2F161395%2F1 "2025-07-01T22:47:39Z")
I am attempting to use AutoTrain Advanced to finetune Mistral-7B-Instruct-v0.2 using a dataset of 2.2 million samples in JSONL format. The dataset is 6.5 GB. I am using Nvidia 8xL40S as the space hardware, the most powerful option available. Partway through generating the test split, before training begins, I receive these errors
```auto
...
Generating train split: 355839 examples [09:57, 466.04 examples/s]
Generating train split: 356339 examples [09:57, 638.27 examples/s]
Generating train split: 356831 examples [09:59, 471.38 examples/s]
Generating train split: 357342 examples [09:59, 637.19 examples/s]
[rank2]:[W701 21:58:39.480892597 socket.cpp:428] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
[rank1]:[W701 21:58:39.491554272 socket.cpp:428] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
[rank7]:[W701 21:58:39.584430192 socket.cpp:428] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
[rank6]:[W701 21:58:39.656034372 socket.cpp:428] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
[rank5]:[W701 21:58:39.666387163 socket.cpp:428] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
[rank3]:[W701 21:58:39.780078100 socket.cpp:428] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
[rank4]:[W701 21:58:39.872609585 socket.cpp:428] [c10d] While waitForInput, poolFD failed with (errno: 0 - Success).
Generating train split: 357808 examples [10:01, 465.60 examples/s]
Generating train split: 358336 examples [10:01, 648.42 examples/s]
Generating train split: 358798 examples [10:02, 466.83 examples/s]
Generating train split: 359000 examples [10:02, 525.05 examples/s]
Generating train split: 359420 examples [10:04, 396.80 examples/s]
Generating train split: 360000 examples [10:04, 607.53 examples/s]
...
```
Shortly after, the space automatically paused. When I attempted to restart the space, I received this error.
I assume that both of these issues are due to the large size of the dataset. I have made it to the beginning of the training step using only the first 500k samples in the dataset, so the issue begins somewhere between 500k and 2.2 million. Is my assumption correct, or could the space have paused for some other reason? Below are the fine-tuning parameters used, they are all at their default values besides merge\_adapter, which is set to True.
```auto
{
"auto_find_batch_size": "false",
"chat_template": "none",
"disable_gradient_checkpointing": "false",
"distributed_backend": "ddp",
"eval_strategy": "epoch",
"merge_adapter": "true",
"mixed_precision": "fp16",
"optimizer": "adamw_torch",
"peft": "true",
"padding": "right",
"quantization": "int4",
"scheduler": "linear",
"unsloth": "false",
"use_flash_attention_2": "false",
"batch_size": "2",
"block_size": "1024",
"epochs": "3",
"gradient_accumulation": "4",
"lr": "0.00003",
"logging_steps": "-1",
"lora_alpha": "32",
"lora_dropout": "0.05",
"lora_r": "16",
"max_grad_norm": "1",
"model_max_length": "2048",
"save_total_limit": "1",
"seed": "42",
"warmup_ratio": "0.1",
"weight_decay": "0",
"target_modules": "all-linear"
}
```
Are there any steps I could take to still use AutoTrain with this large of a dataset? Is it possible to pre-tokenize the dataset, enable streaming, or anything similar? Is there a better way than AutoTrain Advanced to fine-tune a model on a very large dataset (6.5 GB)? Please let me know if I can provide any additional information that would help. Thanks!
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [July 2, 2025, 7:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fautotrain-advanced-error-on-massive-dataset%2F161395%2F2 "2025-07-02T07:50:21Z")
It would be easier if Streaming (`IterableDataset`) could be used, but I don’t know how to enable Streaming in AutoTrainAdvanced…
It might be quicker to split the dataset into several smaller datasets in advance.😅
> [@Best practices for a large dataset](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-practices-for-a-large-dataset%2F137632)%3A
>
> Hi, I have a ~1TB large dataset stored at the HF hub. I can download this on my disk and read it successfully. Nevertheless, it’s large enough that I can’t fit it in my RAM. What is the best practice for training a model on such a dataset? I tried loading the dataset with load\_dataset(..., streaming=True) and then having two buffers: one that is being loaded by training process onto GPU and one that, in a separate thread, streams from the dataset and fills up the other buffer. Then, when the i…
> **[LLM Finetuning with AutoTrain Advanced](https://huggingface.co/docs/autotrain/main/en/tasks/llm_finetuning)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
>
>
> Some datasets require extra arguments to be passed such \`path\` and \`name\` both b…eing present, or \`revision\` etc. Streaming can also be much for efficient for some datasets. It would be great if we could pass some of these as command line arguments, i.e. : \`autotrain llm --data-path=some-data-set --data-kwargs "streaming=true,revision=v1.2" ...\`.
> [@Load Dataset and Save as Parquet](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-dataset-and-save-as-parquet%2F94003)%3A
>
> I am curious whether we can directly save a dataset as parquet as in the original dataset at HF. For example [fineweb-edu/sample/10BT](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu/tree/main/sample/10BT) has 13 parquet files # download the dataset fw = load\_dataset( "HuggingFaceFW/fineweb-edu", name="sample-10BT, split="train", num\_proc=10, cache\_dir=cache\_dir, ) fw.to\_parquet(parquet\_dir) Loads these files as arrow although they are originally parquet, and when saving with to\_parquet() it converts arrow to a single parquet file. So my que…