# Sharing ArrowDataset with subfolders **URL:** **Author:** ![samirchar](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@samirchar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsamirchar)%5C **Post date:** [March 10, 2025, 12:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsharing-arrowdataset-with-subfolders%2F145021%2F1 "2025-03-10T12:41:50Z") Hello everyone! I want to share multiple datasets in the same repo \/\, each in its own folder. The datasets in each folder are already in **sharded** Arrow format (for best performance) and contain different splits, as usual. To read any of these datasets with load\_dataset I would need a loading script to tell HF how to read from the folders, right? If so, should I use the ArrowBasedBuilder and how? I only see tutorials for GeneratorBaseBuilder! Thanks! ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C **Post date:** [March 10, 2025, 3:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsharing-arrowdataset-with-subfolders%2F145021%2F2 "2025-03-10T15:20:45Z")
If it’s already been converted to a Dataset class, is **datasets.concatenate\_dataset** sufficient…? @lhoestq > [@Issue concatenating datasets](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-concatenating-datasets%2F28743)%3A > > I am trying to concatenate two datasets from datasets import load\_dataset, concatenate\_datasets movie = load\_dataset("movie\_rationales") imdb = load\_dataset("imdb") imdb = imdb['train'] Then I adapt the movie dataset movie\_imdb\_format = movie['train'].map( lambda x: {"text": x["review"]} ) movie\_imdb\_format = movie\_imdb\_format.remove\_columns(["review", "evidences"]) and aim to concatenate them dataset\_cc = concatenate\_datasets([imdb, movie\_imdb\_format]) These both datasets output Dat… > [@ArrowBasedBuilder versus GeneratorDBasedBuilder](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Farrowbasedbuilder-versus-generatordbasedbuilder%2F29423)%3A > > Could you please enumerate pros and cons for both these dataset builder classes. I couldn’t find anything in the documentation. When would I prefer one over the other. Is ArrowBasedBuilder more performant for large datasets? Thank you! ---
**Author:** ![samirchar](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@samirchar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsamirchar)%5C **Post date:** [March 10, 2025, 5:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsharing-arrowdataset-with-subfolders%2F145021%2F3 "2025-03-10T17:21:11Z")
@John6666 no because i dont want to concateneate the datasets! Each folder is a different dataset with different features. So do i need the arrow builder to tell HF how to load the different datasets from the subfolder? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C **Post date:** [March 10, 2025, 5:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsharing-arrowdataset-with-subfolders%2F145021%2F4 "2025-03-10T17:34:46Z")
Hmm… In that case, I thought that it would be easier for Hugging Face, which is based on one model per repo (and dataset), to work properly if the datasets with different structures were kept separate.🤔 However, I think there was a way to merge datasets with different structures. Let’s wait for lhonestq. ---
**Author:** ![samirchar](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@samirchar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsamirchar)%5C **Post date:** [March 10, 2025, 6:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsharing-arrowdataset-with-subfolders%2F145021%2F5 "2025-03-10T18:33:00Z")
Yeah, maybe. I’m hesitating to separate into different repos because the datasets are related. It’s not completely separate projects. Think of it as GLUE, which is a set of multiple datasets but they are all related to one objective or project, like shown here [Create a dataset loading script](https://huggingface.co/docs/datasets/en/dataset_script) ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C **Post date:** [March 10, 2025, 11:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsharing-arrowdataset-with-subfolders%2F145021%2F6 "2025-03-10T23:20:32Z")
You can configure the subsets present in your dataset repository in YAML 🙂 see the docs at [Manual Configuration](https://huggingface.co/docs/hub/en/datasets-manual-configuration) See the GLUE dataset for example: [nyu-mll/glue at main](https://huggingface.co/datasets/nyu-mll/glue/tree/main) ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C **Post date:** [March 11, 2025, 3:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsharing-arrowdataset-with-subfolders%2F145021%2F7 "2025-03-11T03:04:10Z")
Thank you! ---
**Author:** ![samirchar](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@samirchar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsamirchar)%5C **Post date:** [March 11, 2025, 11:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsharing-arrowdataset-with-subfolders%2F145021%2F8 "2025-03-11T11:01:53Z")
This is amazing! Thank you very much. ---
**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C **Post date:** [March 11, 2025, 11:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsharing-arrowdataset-with-subfolders%2F145021%2F9 "2025-03-11T23:02:14Z")
Closed on March 11, 2025, 11:02pm UTC This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.