# Dataset subsets with default Dataloader

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-subsets-with-default-dataloader%2F24890%3E%5C
**Category:** 🤗Datasets\
**Created:** [October 24, 2022, 2:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-subsets-with-default-dataloader%2F24890 "2022-10-24T14:44:29Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dennlinger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dennlinger/32/2779_2.png) [@dennlinger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdennlinger)%5C
**Post date:** [October 24, 2022, 2:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-subsets-with-default-dataloader%2F24890%2F1 "2022-10-24T14:44:29Z")

</div>

Hi there!

I was currently working on uploading a [new data resource](https://huggingface.co/datasets/dennlinger/eur-lex-sum), for which I have 24 languages as subsets, each with their respective JSONL-formatted data splits (train/validation/test).  
Given that I a great experience with using the default data loader in previous datasets, I wanted to rely on it again in this case; however, it seems that despite my attempts at structuring the `data/` folder by language, it is not automatically recognized as a respective subset. Quite the opposite, the data is mixed across all folders into one giant train/validation/test portion.

I have tried to look at how other multilingual datasets structure their data, and it seems all of them provide their custom loader scripts.  
My question is now whether there is a way to make the default loader accept the folder strucutre, and provide access to dedicated subsets? Or do I have to rely on my own custom scripts in such an instance?

Many thanks in advance for any pointers!  
Best,  
Dennis

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [October 24, 2022, 3:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-subsets-with-default-dataloader%2F24890%2F2 "2022-10-24T15:49:28Z")

</div>

Hi! Yes, a custom script is needed at the moment to define multiple subsets/configs. [Add support to create different configs with `push_to_hub` (+ inferring configs from directories with package managers?) · Issue #5151 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/issues/5151) should implement the automatic inference of configs, so feel free to comment on this issue if you have some suggestions or/and subscribe to track progress.

---

<div class="post-metadata">

**Author:** ![dennlinger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dennlinger/32/2779_2.png) [@dennlinger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdennlinger)%5C
**Post date:** [October 25, 2022, 12:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-subsets-with-default-dataloader%2F24890%2F3 "2022-10-25T12:00:55Z")

</div>

Alright, thanks for the fast reply! Glad to see the issue, I’ll stop by to check what the current proposal looks like!
