# Add a subset to a dataset from CLI?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-a-subset-to-a-dataset-from-cli%2F139152%3E%5C
**Category:** 🤗Datasets\
**Created:** [February 5, 2025, 3:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-a-subset-to-a-dataset-from-cli%2F139152 "2025-02-05T15:19:26Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![geoffnn](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/geoffnn/32/37301_2.png) [@geoffnn](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgeoffnn)%5C
**Post date:** [February 5, 2025, 3:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-a-subset-to-a-dataset-from-cli%2F139152%2F1 "2025-02-05T15:19:26Z")

</div>

Hi!  
I’m trying to modify a dataset on HF, that contains subsets – essentially processing it and re-uploading it. e.g. this one [Salesforce/lotsa\_data · Datasets at Hugging Face](https://huggingface.co/datasets/Salesforce/lotsa_data/)  
What I’m doing currently is download the parquet for each subset, process it independently, store it in an S3 folder, and then use the CLI to upload the folder. This does not result in a dataset with subsets…  
What’s the fastest way of reproducing the structure of the initial dataset after processing?

I think this is related to [/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcreate-multiple-dataset-subsets-at-the-same-time%2F12997%5D(https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcreate-multiple-dataset-subsets-at-the-same-time%2F12997)  
Thanks,

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [February 5, 2025, 3:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-a-subset-to-a-dataset-from-cli%2F139152%2F2 "2025-02-05T15:26:09Z")

</div>

The latter is in Japanese, so it may be a little difficult to read. Google translate it…  
I think that in the past, creating a loading\_script was the best thing to do. Now, it may be smarter to use a custom DatasetBuilder.

> [@How to create subset when pushing to hub](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-subset-when-pushing-to-hub%2F19542)%3A
>
> Hey! I have a dataset of image and text, and I am trying to upload it to the hub using the script below. I was wondering how to create a subset, because everything is been put in “gabrielsantosrv–pracegover” subset. Thanks in advance smiley from datasets import Dataset, load\_dataset from PIL import Image import io import json def try\_load\_image(filepath): try: with open(filepath, 'rb') as f: image = Image.open(io.BytesIO(f.read())) if isinstance(image, Image…

> **[HuggingFace datasetsのBuilder classを使ってデータセットを自作する](https://zenn.dev/mjun0812/articles/cfe4b7346ba6b4)**