# Torchrun, trainer, dataset setup
**URL:**
**Author:**  [@wdavies](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwdavies)%5C
**Post date:** [December 8, 2024, 9:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftorchrun-trainer-dataset-setup%2F130026%2F1 "2024-12-08T21:35:46Z")
Hi,
I have a piece of python code, that loads a dataset from a file, splits it into train test, and then sets up a Trainer with the two splits as the train and validation sets. I am training a distilBERT model (I dont think that detail matters).
It works great. Even more so, I can do torchrun --nproc\_per\_node = 4, and it seems to run great too, it _seems_ to spawn 4 versions and trains them on the 4 GPUs.
However, I just got suspicious - given that I’m doing a train test split every time the python code spins up (one per GPU - I can see it by printing out some debug), is the Trainer basically training on just the first 25% of each training set (and ditto for validation).
I’d assumed initially it was smarter than this, and some how only used one of the train/test instances (presumably the first), but now I realize I cannot be certain of this. So, can anyone tell me if I need to “pre-partition” the datasets into 4 shards, and then load them keyed by LOCAL RANK?
Thanks in advance,
W
---
**Author:**  [@wdavies](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwdavies)%5C
**Post date:** [December 8, 2024, 9:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftorchrun-trainer-dataset-setup%2F130026%2F2 "2024-12-08T21:37:44Z")
Also, less importantly, the train\_test split may be non deterministic, so that even if there was some kind of hash to tell that each worker had the same train and validation inputs and to just shard them, the fact that I’ve potentially got a different train/test split for each evocation of the python code, MIGHT, confuse things.
---
**Author:**  [@wdavies](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwdavies)%5C
**Post date:** [December 12, 2024, 11:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftorchrun-trainer-dataset-setup%2F130026%2F3 "2024-12-12T23:39:47Z")
So it looks like noone knows. I wish there was a way to get an actual huggingface employee to answer this question. Unfortunately I don’t think pytorch community will know.
[https://pytorch.org/docs/stable/elastic/run.html](https://pytorch.org/docs/stable/elastic/run.html)
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [December 13, 2024, 3:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftorchrun-trainer-dataset-setup%2F130026%2F4 "2024-12-13T03:30:03Z")
I think trainer is part of the transformers library, so I think you can check it by opening an issue on the transformers github.
> **[Issues · huggingface/transformers](https://github.com/huggingface/transformers/issues)**
>
> 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX. - Issues · huggingface/transformers
---
**Author:**  [@wdavies](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwdavies)%5C
**Post date:** [December 20, 2024, 5:12pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftorchrun-trainer-dataset-setup%2F130026%2F5 "2024-12-20T17:12:27Z")
Here’s the open issue:
>
>
> \### System Info
>
> \- \`transformers\` version: 4.47.0
> \- Platform: Linux-6.8.0-51-ge…neric-x86\_64-with-glibc2.35
> \- Python version: 3.11.6
> \- Huggingface\_hub version: 0.26.3
> \- Safetensors version: 0.4.5
> \- Accelerate version: 1.1.1
> \- Accelerate config: not found
> \- PyTorch version (GPU?): 2.0.1+cu117 (True)
> \- Tensorflow version (GPU?): 2.15.1 (True)
> \- Flax version (CPU?/GPU?/TPU?): not installed (NA)
> \- Jax version: not installed
> \- JaxLib version: not installed
> \- Using distributed or parallel set-up in script?: CUDA\_VISIBLE\_DEVICES=0,1,2,3 torchrun --nproc-per-node 4 stages/train\_model.py
> \- Using GPU in script?: Yes
> \- GPU type: NVIDIA RTX A4000
>
> \### Who can help?
>
> @muellerzr @SunMarc
>
> I have a piece of python code, that loads a training dataset from a file, then sets up a Trainer with the training dataset. I am training a distilBERT model (I dont think that detail matters).
>
> It works great. And then I can do torchrun --nproc\_per\_node = 4, and it seems to run great too, it seems to spawn 4 versions and trains them on the 4 GPUs.
>
> However, I got suspicious - given that I’m passing the same dataset in everyt ime, is the Trainer basically training on just the first 25% of the training set?
>
> I’d assumed initially it was smarter than this, and some how only used one of the train dataset instances (presumably the first), but now I realize I cannot be certain of this. So, can anyone tell me if I need to “pre-partition” the datasets into 4 shards, and then load them keyed by LOCAL RANK?
>
> If it's not doing the "right" thing - i.e. training on all the data, then a warning should be given when this is detected.
>
> Script looks like this:
> train\_test\_dataset = Dataset.load\_from\_disk('train\_test\_dataset\_file')
> trainer = Trainer(..., train\_dataset=train\_test\_dataset\['train'\], validation\_dataset=train\_test\_dataset\['test'\],...)
>
>
> Should script looks like this:
> train\_test\_dataset = Dataset.load\_from\_disk(f"one\_quarter\_train\_test\_dataset\_file\_{LOCAL\_RANK)")
> trainer = Trainer(..., train\_dataset=train\_test\_dataset\['train'\], validation\_dataset=train\_test\_dataset\['test'\],...)
>
>
>
> \### Information
>
> \- \[\] The official example scripts
> \- \[X\] My own modified scripts
>
> \### Tasks
>
> \- \[\] An officially supported task in the \`examples\` folder (such as GLUE/SQuAD, ...)
> \- \[X\] My own task or dataset (give details below)
>
> \### Reproduction
>
> clarity is needed on what behaviour is happening with the training data. Is the training dataset being used by all the nodes with an equal division of labor over each quarter of the dataset (or interleaved every 4 minibatches), or is it sending the same minibatch from the training set, and never getting to the final three quarters of the dataset.
>
> \### Expected behavior
>
> A better understanding of what data is being trained on when invoking Trainer through torchrun.