# How to steaming .hf dataset
**URL:**
**Author:**  [@xDokiXDokix](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FxDokiXDokix)%5C
**Post date:** [November 6, 2024, 1:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-steaming-hf-dataset%2F116008%2F1 "2024-11-06T13:45:03Z")
I’m trying to load .hf datasets using stream
The dataset is tuan124816/newcs2\_data
```auto
dataset = load_dataset("tuan124816/newcs2_data",
streaming=True)
hf_dataset = dataset['test']
```
output
```auto
IterableDataset({
features: Unknown,
n_shards: 600
})
```
when print out the first element:
```auto
print(next(iter(hf_dataset)))
```
output
```auto
{'_data_files': [{'filename': 'data-00000-of-00001.arrow'}], '_fingerprint': '905978a8bab44335', '_format_columns': ['observations', 'actions', 'rewards'], '_format_kwargs': {}, '_format_type': None, '_output_all_columns': False, '_split': None}
```
Is this the right way to load this kind of dataset?
How can I read the data and know what inside [‘observations’, ‘actions’, ‘rewards’]?
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [November 6, 2024, 3:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-steaming-hf-dataset%2F116008%2F2 "2024-11-06T15:20:55Z")
This?
> **[Differences between Dataset and IterableDataset](https://huggingface.co/docs/datasets/about_mapstyle_vs_iterable)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
---
**Author:**  [@xDokiXDokix](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FxDokiXDokix)%5C
**Post date:** [November 6, 2024, 3:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-steaming-hf-dataset%2F116008%2F3 "2024-11-06T15:54:14Z")
From what I see , the stream only load the state.json file from each .hf folder
state.json:
```auto
{
"_data_files": [
{
"filename": "data-00000-of-00001.arrow"
}
],
"_fingerprint": "905978a8bab44335",
"_format_columns": [
"observations",
"actions",
"rewards"
],
"_format_kwargs": {},
"_format_type": null,
"_output_all_columns": false,
"_split": null
}
```
From my experience other dataset like imdb when stream alway have an clear output of dictionary with text and label. I’m confuse why the data don’t load the arrow file.
Am I doing something wrong?
---
**Author:**  [@xDokiXDokix](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FxDokiXDokix)%5C
**Post date:** [November 10, 2024, 3:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-steaming-hf-dataset%2F116008%2F4 "2024-11-10T15:13:45Z")
I fix this , because of the dataset viewer on hugging face
I don’t know why but when I delete it from dataset then the stream work fine again.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [November 10, 2024, 3:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-steaming-hf-dataset%2F116008%2F5 "2024-11-10T15:34:00Z")
Ah, it’s probably a bug in the library.
There are reports that sound like they might be, but I don’t think they’ve been merged.
> **[Issues · huggingface/datasets](https://github.com/huggingface/datasets/issues)**
>
> 🤗 The largest hub of ready-to-use datasets for ML models with fast, easy-to-use and efficient data manipulation tools - Issues · huggingface/datasets
[https://github.com/huggingface/datasets/pulls](https://github.com/huggingface/datasets/pulls)
Does the behavior of the dataset viewer affect the behavior of the dataset library?
Does it lock the file or something…
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [November 30, 2024, 4:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-steaming-hf-dataset%2F116008%2F6 "2024-11-30T16:13:24Z")
Hi ! Datasets with a state.json are datasets saved for local disk using `save_to_disk()` and are not supported by the Hub - though if you remove the json file to only keep the Arrow files it will work correctly.
We should change `.save_to_disk()` completely to save in a format that is supported by the Hub IMO