# Streaming Wikipedia dataset

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-wikipedia-dataset%2F35747%3E%5C
**Category:** 🤗Datasets\
**Created:** [April 6, 2023, 9:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-wikipedia-dataset%2F35747 "2023-04-06T09:29:46Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![alvations](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alvations/32/97_2.png) [@alvations](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falvations)%5C
**Post date:** [April 6, 2023, 9:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-wikipedia-dataset%2F35747%2F1 "2023-04-06T09:29:47Z")

</div>

When using the `load_dataset('wikipedia', '20220301.en')` dataset with `streaming=True`, it throws a ValueError:

```auto
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-4-af223b124995> in <cell line: 1>()
----> 1 load_dataset('wikipedia', lang, streaming=True)

1 frames
/usr/local/lib/python3.9/dist-packages/datasets/builder.py in as_streaming_dataset(self, split, base_path)
   1247 ) -> Union[Dict[str, IterableDataset], IterableDataset]:
   1248 if not isinstance(self, (GeneratorBasedBuilder, ArrowBasedBuilder)):
-> 1249 raise ValueError(f"Builder {self.name} is not streamable.")
   1250 
   1251 is_local = not is_remote_filesystem(self._fs)

ValueError: Builder wikipedia is not streamable.

```

And to “work-around” it, it seems a little meta (fourth-wall), and this works:

```auto

from datasets import load_dataset, IterableDataset

from torch.utils.data import DataLoader
from torchdata.datapipes.iter import IterDataPipe, IterableWrapper

# Load from HF.
_ds = load_dataset('wikipedia', '20220301.en')

def _ds_gen():
    for i in range(len(_ds)):
        yield _ds['train'][i]

dataloader = DataLoader(
    IterableDataset.from_generator(_ds_gen)
)

```

**Is there any way to add above code that generator wrapper over the `load_dataset('wikipedia', '20220301.en', streaming=True)`?**

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [April 6, 2023, 1:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-wikipedia-dataset%2F35747%2F2 "2023-04-06T13:33:14Z")

</div>

Hi! We are working on making the `wikipedia` dataset streamable in this PR: [Support streaming Beam datasets from HF GCS preprocessed data by albertvillanova · Pull Request #5689 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/pull/5689)

---

<div class="post-metadata">

**Author:** ![alvations](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alvations/32/97_2.png) [@alvations](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falvations)%5C
**Post date:** [April 7, 2023, 3:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-wikipedia-dataset%2F35747%2F3 "2023-04-07T03:33:15Z")

</div>

Thanks for the prompt reply! I guess for now, we have to stream the dataset with the “meta-snippet”.