# Sharing ArrowDataset with subfolders

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsharing-arrowdataset-with-subfolders%2F145021%3E%5C
**Category:** 🤗Datasets\
**Created:** [March 10, 2025, 12:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsharing-arrowdataset-with-subfolders%2F145021 "2025-03-10T12:41:49Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [March 10, 2025, 3:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsharing-arrowdataset-with-subfolders%2F145021%2F2 "2025-03-10T15:20:45Z")

</div>

If it’s already been converted to a Dataset class, is **datasets.concatenate\_dataset** sufficient…? @lhoestq

> [@Issue concatenating datasets](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissue-concatenating-datasets%2F28743)%3A
>
> I am trying to concatenate two datasets from datasets import load\_dataset, concatenate\_datasets movie = load\_dataset("movie\_rationales") imdb = load\_dataset("imdb") imdb = imdb['train'] Then I adapt the movie dataset movie\_imdb\_format = movie['train'].map( lambda x: {"text": x["review"]} ) movie\_imdb\_format = movie\_imdb\_format.remove\_columns(["review", "evidences"]) and aim to concatenate them dataset\_cc = concatenate\_datasets([imdb, movie\_imdb\_format]) These both datasets output Dat…

> [@ArrowBasedBuilder versus GeneratorDBasedBuilder](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Farrowbasedbuilder-versus-generatordbasedbuilder%2F29423)%3A
>
> Could you please enumerate pros and cons for both these dataset builder classes. I couldn’t find anything in the documentation. When would I prefer one over the other. Is ArrowBasedBuilder more performant for large datasets? Thank you!

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsharing-arrowdataset-with-subfolders%2F145021)._