# Save \`DatasetDict\` to HuggingFace Hub
**URL:**
**Author:**  [@pietrolesci](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpietrolesci)%5C
**Post date:** [November 19, 2021, 7:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsave-datasetdict-to-huggingface-hub%2F12075%2F1 "2021-11-19T19:17:18Z")
Hi there,
I prepared my data into a `DatasetDict` object that I saved to disk with the `save_to_disk` method. Iâd like to upload the generated folder to the HuggingFace Hub and use it using the usual `load_dataset` function. Though, I have not yet found a way to do so. Is this possible?
Thanks a lot in advance for your help.
Best,
Pietro
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [November 22, 2021, 3:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsave-datasetdict-to-huggingface-hub%2F12075%2F2 "2021-11-22T15:35:59Z")
Hi,
this weekâs release of `datasets` will add support for directly pushing a `Dataset`/`DatasetDict` object to the Hub. In the meantime, you can use a `to_{format}` method, where format is one of `["csv", "json", "txt", "parquet"]` on each split of the `DatasetDict` object and push the generated files to the Hub (follow the docs [here](https://huggingface.co/docs/datasets/master/upload_dataset.html) for more information). Also note that this requires the master version of the library, which you can install with:
```bash
pip install git+https://github.com/huggingface/datasets.git
```
Without the master version, youâll have to specify a list of files to load each split separately (docs on that are [here](https://huggingface.co/docs/datasets/v1.15.1/upload_dataset.html)).
---
**Author:**  [@pietrolesci](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpietrolesci)%5C
**Post date:** [November 23, 2021, 8:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsave-datasetdict-to-huggingface-hub%2F12075%2F3 "2021-11-23T20:55:32Z")
Hi @mariosasko,
Thanks a lot for your answer! I will try this out later and let you know how it goes. Excited about the new upcoming feature đ
Best,
Pietro
---
**Author:**  [@pierreguillou](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpierreguillou)%5C
**Post date:** [December 6, 2021, 6:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsave-datasetdict-to-huggingface-hub%2F12075%2F4 "2021-12-06T18:46:10Z")
> [@mariosasko](#):
>
> this weekâs release of `datasets` will add support for directly pushing a `Dataset` / `DatasetDict` object to the Hub.
Hi @mariosasko,
I just followed the guide [Upload from Python](https://huggingface.co/docs/datasets/master/upload_dataset.html#upload-from-python) to push to the datasets hub a `DatasetDict` with train and validation `Datasets` inside.
```auto
raw_datasets = DatasetDict({
train: Dataset({
features: ['translation'],
num_rows: 10000000
})
validation: Dataset({
features: ['translation'],
num_rows: 1000000
})
})
from huggingface_hub import notebook_login
notebook_login()
raw_datasets.push_to_hub(repo_id=dataset_name, private=True)
```
The [DatasetDict.push\_to\_hub()](https://huggingface.co/docs/datasets/master/package_reference/main_classes.html#datasets.DatasetDict.push_to_hub) works, and I have train and validation `parquet` files in my repository (in the folder `data`) but when I do a `load_dataset()`, I got a `DatasetDict` with only a `Dataset` train that has all the rows (11000000) from the original `Dataset` train (10000000) and `Dataset` validation (1000000) that were pushed.
```auto
import datasets
from datasets import load_dataset
raw_datasets = load_dataset(dataset_name, use_auth_token=True)
raw_datasets
DatasetDict({
train: Dataset({
features: ['translation'],
num_rows: 11000000
})
})
```
Strange. How can I get my original `DatasetDict` with `load_dataset()`? Thanks.
---
**Author:**  [@pierreguillou](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpierreguillou)%5C
**Post date:** [December 6, 2021, 7:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsave-datasetdict-to-huggingface-hub%2F12075%2F5 "2021-12-06T19:23:40Z")
@mariosasko, I guess my problem of concatenation between train and validation when using `load_dataset()` seems to be a normal behavior if no [dataset loading script](https://huggingface.co/docs/datasets/dataset_script.html) is created in the files on the HF dataset repository (check this [post](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcreating-and-uploading-dataset-huggingface-hub-vs-dataset-creation-script%2F11078%2F2))%3A
> [@lhoestq](#):
>
> If you donât upload a dataset script, then the default dataset builder for .txt file is used (and basically it concatenates all the text data together).
However, this information is not given in the HF doc [Upload from Python](https://huggingface.co/docs/datasets/master/upload_dataset.html#upload-from-python) about how to upload a [`datasets.DatasetDict`](https://huggingface.co/docs/datasets/master/package_reference/main_classes.html#datasets.DatasetDict) on the Hugging Face Hub in Python.
The HF doc has to be updated or the [DatasetDict.push\_to\_hub()](https://huggingface.co/docs/datasets/master/package_reference/main_classes.html#datasets.DatasetDict.push_to_hub) has to be modified?
cc @lhoestq
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [December 7, 2021, 11:31am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsave-datasetdict-to-huggingface-hub%2F12075%2F6 "2021-12-07T11:31:33Z")
Hi ! Since `push_to_hub` has been introduced, the dataset builder doesnât concatenate everything anymore. Indeed now it takes into account the file names. Each file containing âvalidationâ goes to the validation split for example. The supported patterns are explained in the [docs](https://huggingface.co/docs/datasets/repository_structure.html)
Make sure to update `datasets` to make sure you load each split independently đ
---
**Author:**  [@pierreguillou](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpierreguillou)%5C
**Post date:** [December 7, 2021, 12:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsave-datasetdict-to-huggingface-hub%2F12075%2F7 "2021-12-07T12:14:14Z")
Hi @lhoestq,
> [@lhoestq](#):
>
> Make sure to update `datasets`
You are right.
I was using the [HF sagemaker notebooks](https://github.com/huggingface/notebooks/tree/master/sagemaker) where the datasets version is inferior to the [1.16](https://github.com/huggingface/datasets/releases/tag/1.16.0) (often 1.13, cc @philschmid) where the [DatasetDict.push\_to\_hub()](https://huggingface.co/docs/datasets/master/package_reference/main_classes.html#datasets.DatasetDict.push_to_hub) method was introduced.
With this version (and the latest one [1.16.1](https://github.com/huggingface/datasets/releases/tag/1.16.1)), it works (I get the same `DatasetDict` with train and validation that the one I pushed to the HF datasets hub):
```auto
# !pip install datasets --upgrade
import datasets
print(datasets. __version__ )
# 1.16.1
from datasets import load_dataset
raw_datasets = load_dataset(dataset_name, use_auth_token=True)
raw_datasets
raw_datasets = DatasetDict({
train: Dataset({
features: ['translation'],
num_rows: 10000000
})
validation: Dataset({
features: ['translation'],
num_rows: 1000000
})
})
```
---
**Author:**  [@pierreguillou](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpierreguillou)%5C
**Post date:** [December 10, 2021, 1:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsave-datasetdict-to-huggingface-hub%2F12075%2F8 "2021-12-10T01:58:10Z")
Hi @lhoestq,
As already discussed, we need to use `datasets>=1.16` in order to `push_to_hub()` and `load_dataset()` a `DatasetDict()`. This is very clear.
However, I checked if the `features` of the `DatasetDict()` are kept and it appears that not.
```auto
# download https://huggingface.co/datasets/lener_br
datasets = load_dataset('lener_br')
# check the features
datasets['train'].features
{'id': Value(dtype='string', id=None),
'ner_tags': Sequence(feature=ClassLabel(num_classes=13, names=['O', 'B-ORGANIZACAO', 'I-ORGANIZACAO', 'B-PESSOA', 'I-PESSOA', 'B-TEMPO', 'I-TEMPO', 'B-LOCAL', 'I-LOCAL', 'B-LEGISLACAO', 'I-LEGISLACAO', 'B-JURISPRUDENCIA', 'I-JURISPRUDENCIA'], names_file=None, id=None), length=-1, id=None),
'tokens': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
# connect to HF hub
from huggingface_hub import notebook_login
notebook_login()
# push this DatasetDict() to my HF profile as private
datasets.push_to_hub(repo_id='test_lener_br', private=True)
# download the pushed DatasetDict()
datasets = load_dataset('pierreguillou/test_lener_br', use_auth_token=API_TOKEN)
# check the features
datasets['train'].features
{'id': Value(dtype='string', id=None),
'ner_tags': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None),
'tokens': Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)}
```
**As you can see, I lost the ner\_tags `features` of the original `DatasetDict()`.** What do you think?
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [December 10, 2021, 11:02am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsave-datasetdict-to-huggingface-hub%2F12075%2F9 "2021-12-10T11:02:23Z")
It looks like a bug, thanks for reporting !
I think others have had issues with ClassLabel recently (see [here](https://github.com/huggingface/datasets/issues/3394)). Weâre investigating whatâs going on đ
---
**Author:**  [@stepkurniawan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fstepkurniawan)%5C
**Post date:** [October 19, 2023, 12:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsave-datasetdict-to-huggingface-hub%2F12075%2F10 "2023-10-19T12:54:21Z")
Sorry, I still donât get the answerâŚ
I have some dataset that i save using save\_to\_disk()
how to upload that to HF with a nice subset name?
đ
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [October 19, 2023, 2:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsave-datasetdict-to-huggingface-hub%2F12075%2F11 "2023-10-19T14:04:07Z")
@stepkurniawan You can use `config_name` to assign the name to a subset (e.g. `dataset_dict.push_to_hub(repo_id, config_name=config_name, ...)`)
---
**Author:**  [@stepkurniawan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fstepkurniawan)%5C
**Post date:** [October 19, 2023, 6:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsave-datasetdict-to-huggingface-hub%2F12075%2F12 "2023-10-19T18:42:19Z")
Hi Mario,
Thank you for answering.
Yes, and when I tried to load\_dataset( âstepkurniawan/xxxâ, â\â ) , the issue arises.
it somehow only returns me my old dataset that I have uploaded in the first time. see: [Load\_dataset() doesnât load ONE of the Subset - Beginners - Hugging Face Forums](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-dataset-doesnt-load-one-of-the-subset%2F58874)
Please tell me when im wrong.
If you use push\_to\_hub(âsomethingâ, config\_name=âconfignameâ), when it exists already. Will the second time overwrites the first push?
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [October 20, 2023, 10:12am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsave-datasetdict-to-huggingface-hub%2F12075%2F13 "2023-10-20T10:12:19Z")
> [@stepkurniawan](#):
>
> If you use push\_to\_hub(âsomethingâ, config\_name=âconfignameâ), when it exists already. Will the second time overwrites the first push?
Yes it overwrites it