# Problem in pushing dataset to the Hub

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fproblem-in-pushing-dataset-to-the-hub%2F61090%3E%5C
**Category:** 🤗Datasets\
**Created:** [November 5, 2023, 2:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fproblem-in-pushing-dataset-to-the-hub%2F61090 "2023-11-05T02:50:36Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![murodbek](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/murodbek/32/29433_2.png) [@murodbek](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmurodbek)%5C
**Post date:** [November 5, 2023, 2:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fproblem-in-pushing-dataset-to-the-hub%2F61090%2F1 "2023-11-05T02:50:36Z")

</div>

I am trying to add Turkic Machine Translation dataset that is introduced by this [paper](https://arxiv.org/pdf/2109.04593) and stored in [there](https://drive.google.com/drive/folders/1kUp_vpDsNUZvVC6HvwNxGn7ImwnCfM1E?usp=sharing) to be able to use it easily. I downloaded all sets and using this script to pre-process:

```auto
import os
from tqdm.notebook import tqdm
from datasets import Dataset, DatasetDict

lang_combs = set()
splits = ['train', 'dev', 'test/bible', 'test/ted', 'test/x-wmt']

for split in splits:
    lang_combs.update(os.listdir(split))

lang_combs = sorted(list(lang_combs))

datasets = {}

for lang_comb in tqdm(lang_combs):
    dataset = {}

    for split in splits:
        split_data = {"translation": []}
        src_lang, tgt_lang = lang_comb.split('-')
        src_file = os.path.join(split, lang_comb, f"{src_lang}-{tgt_lang}.{src_lang}")
        tgt_file = os.path.join(split, lang_comb, f"{src_lang}-{tgt_lang}.{tgt_lang}")
        
        if os.path.exists(src_file) and os.path.exists(tgt_file):
            with open(src_file, 'r') as src, open(tgt_file, 'r') as tgt:
                src_sents = [line.strip() for line in src.readlines()]
                tgt_sents = [line.strip() for line in tgt.readlines()]

            for src_sent, tgt_sent in zip(src_sents, tgt_sents):
                split_data["translation"].append({
                    src_lang: src_sent,
                    tgt_lang: tgt_sent
                })

            split = split.replace('/', '-')
            dataset[split] = Dataset.from_dict(split_data)

    dataset_dict = DatasetDict(dataset)
    datasets[lang_comb] = dataset_dict

final_dataset = DatasetDict(datasets)

```

Then, when I tried to push it to the Hub, I am getting this error:

TypeError: Values in `DatasetDict` should be of type `Dataset` but got type ‘\<class ‘datasets.dataset\_dict.DatasetDict’\>’

The authors of original paper have [Turkic X-WMT](https://huggingface.co/datasets/turkic_xwmt) dataset but this generated from script ([turkic\_xwmt.py](https://huggingface.co/datasets/turkic_xwmt/blob/main/turkic_xwmt.py)). And what is the difference between pushing it in parquet files and writing script. As far as I know, it is suitable for the datasets that is accessible via an API. So, how can I solve this?

---

<div class="post-metadata">

**Author:** ![panigrah](https://avatars.discourse-cdn.com/v4/letter/p/bb73d2/32.png) [@panigrah](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpanigrah)%5C
**Post date:** [November 5, 2023, 3:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fproblem-in-pushing-dataset-to-the-hub%2F61090%2F2 "2023-11-05T03:56:59Z")

</div>

I think this is the problem

> <https://github.com/huggingface/datasets/issues/4211#issuecomment-1109461507>
>
> Hi there,
> 
> I am trying to load a dataset to the Hub. This dataset is a \`Datase…tDict\` composed of various splits. Some splits have a different \`Feature\` mapping. Locally, the DatasetDict preserves the individual features but if I \`push\_to\_hub\` and then \`load\_dataset\`, the features are all the same.
> 
> Dataset and code to reproduce available \[here\](https://huggingface.co/datasets/pietrolesci/robust\_nli).
> 
> In short:
> 
> I have 3 feature mapping
> \`\`\`python
> Tri\_features = Features(
> {
> "idx": Value(dtype="int64"),
> "premise": Value(dtype="string"),
> "hypothesis": Value(dtype="string"),
> "label": ClassLabel(num\_classes=3, names=\["entailment", "neutral", "contradiction"\]),
> }
> )
> 
> Ent\_features = Features(
> {
> "idx": Value(dtype="int64"),
> "premise": Value(dtype="string"),
> "hypothesis": Value(dtype="string"),
> "label": ClassLabel(num\_classes=2, names=\["non-entailment", "entailment"\]),
> }
> )
> 
> Con\_features = Features(
> {
> "idx": Value(dtype="int64"),
> "premise": Value(dtype="string"),
> "hypothesis": Value(dtype="string"),
> "label": ClassLabel(num\_classes=2, names=\["non-contradiction", "contradiction"\]),
> }
> )
> \`\`\`
> 
> Then I create different datasets
> 
> \`\`\`python
> dataset\_splits = {}
> 
> for split in df\["split"\].unique():
> print(split)
> df\_split = df.loc\[df\["split"\] == split\].copy()
>     
> if split in Tri\_dataset:
> df\_split\["label"\] = df\_split\["label"\].map({"entailment": 0, "neutral": 1, "contradiction": 2})
> ds = Dataset.from\_pandas(df\_split, features=Tri\_features)
>     
> elif split in Ent\_bin\_dataset:
> df\_split\["label"\] = df\_split\["label"\].map({"non-entailment": 0, "entailment": 1})
> ds = Dataset.from\_pandas(df\_split, features=Ent\_features)
>     
> elif split in Con\_bin\_dataset:
> df\_split\["label"\] = df\_split\["label"\].map({"non-contradiction": 0, "contradiction": 1})
> ds = Dataset.from\_pandas(df\_split, features=Con\_features)
> 
> else:
> print("ERROR:", split)
> dataset\_splits\[split\] = ds
> datasets = DatasetDict(dataset\_splits)
> \`\`\`
> 
> I then push to hub
> 
> \`\`\`python
> datasets.push\_to\_hub("pietrolesci/robust\_nli", token="\<token\>")
> \`\`\`
> 
> Finally, I load it from the hub
> 
> \`\`\`python
> datasets\_loaded\_from\_hub = load\_dataset("pietrolesci/robust\_nli")
> \`\`\`
> 
> And I get that
> 
> \`\`\`python
> datasets\["LI\_TS"\].features != datasets\_loaded\_from\_hub\["LI\_TS"\].features
> \`\`\`
> 
> since 
> 
> \`\`\`python
> "label": ClassLabel(num\_classes=2, names=\["non-contradiction", "contradiction"\])
> \`\`\`
> 
> gets remapped to 
> 
> \`\`\`python
> "label": ClassLabel(num\_classes=3, names=\["entailment", "neutral", "contradiction"\])
> \`\`\`

What you have is a dataset with multiple configurations. Each language combination is a different configuration and I suspect that the upload function still does not support pushing multiple configurations programmatically at the same time.

---

<div class="post-metadata">

**Author:** ![murodbek](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/murodbek/32/29433_2.png) [@murodbek](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmurodbek)%5C
**Post date:** [November 7, 2023, 1:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fproblem-in-pushing-dataset-to-the-hub%2F61090%2F3 "2023-11-07T13:46:21Z")

</div>

Thank you for your response, but I think the problem doesn’t solve with this. What I believe is the problem is that I am trying to push dataset that consisted of 405 **DatasetDict** s that each has `train`, `dev` and `test` **Dataset** s. You see the error message?

TypeError: Values in `DatasetDict` should be of type `Dataset` but got type ‘\<class ‘datasets.dataset\_dict.DatasetDict’\>’

Other translation datasets is created not by pushing all the datasets to the Hub but creating wrapper for original source. So, how can I push this dataset to the hub as parquet files or make a wrapper from Google Drive?

---

<div class="post-metadata">

**Author:** ![panigrah](https://avatars.discourse-cdn.com/v4/letter/p/bb73d2/32.png) [@panigrah](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpanigrah)%5C
**Post date:** [November 9, 2023, 12:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fproblem-in-pushing-dataset-to-the-hub%2F61090%2F4 "2023-11-09T12:26:48Z")

</div>

It’s the same thing from what I understand. Doesn’t support pushing a dataset of dictionaries.

Anyway you can look at using the HFApi as an alternative

> **[Upload files to the Hub](https://huggingface.co/docs/huggingface_hub/guides/upload)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.