# Converting an HF dataset to pandas

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-an-hf-dataset-to-pandas%2F12043%3E%5C
**Category:** Beginners\
**Created:** [November 18, 2021, 9:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-an-hf-dataset-to-pandas%2F12043 "2021-11-18T21:14:10Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![alghar](https://avatars.discourse-cdn.com/v4/letter/a/e68b1a/32.png) [@alghar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falghar)%5C
**Post date:** [November 18, 2021, 9:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-an-hf-dataset-to-pandas%2F12043%2F1 "2021-11-18T21:14:10Z")

</div>

Wondering if there is a way to convert a dataset downloaded using load\_dataset to pandas?

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [November 19, 2021, 11:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-an-hf-dataset-to-pandas%2F12043%2F2 "2021-11-19T11:07:38Z")

</div>

Hi,

we have a method for that - `Dataset.to_pandas`. However, note that this will load the entire dataset into memory by default to create a DataFrame. If your dataset is too big to fit in RAM, load it in chunks as follows:

```python
dset = load_dataset(...)
for df in dset.to_pandas(batch_size=..., batched=True):
    # process dataframes

```

Another option is to use the `pandas` formatter, which will return a DataFrame object each time the dataset is indexed/sliced:

```python
dset = load_dataset(...)
dset.set_format("pandas")
dset[10] # returns a dataframe with 1 row
dset[10:30] # returns a dataframe with 20 rows
```

---

<div class="post-metadata">

**Author:** ![YanaS](https://avatars.discourse-cdn.com/v4/letter/y/57b2e6/32.png) [@YanaS](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FYanaS)%5C
**Post date:** [April 3, 2024, 8:04am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-an-hf-dataset-to-pandas%2F12043%2F3 "2024-04-03T08:04:44Z")

</div>

Just a little add-on: if the Huggingface Dataset consists of train and test data, after the set\_format you should write for example:  
`dataset['train'][10]`  
Otherwise, it returns Invalid key error.

---

<div class="post-metadata">

**Author:** ![vpkprasanna](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vpkprasanna/32/26436_2.png) [@vpkprasanna](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvpkprasanna)%5C
**Post date:** [June 17, 2024, 3:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-an-hf-dataset-to-pandas%2F12043%2F4 "2024-06-17T15:35:38Z")

</div>

> [@mariosasko](#):
>
> ```auto
> for df in dset.to_pandas(batch_size=..., batched=True):
> 
> ```

Got the below error when trying the above method  
AttributeError: ‘DatasetDict’ object has no attribute ‘to\_pandas’