# Streaming Dataset Roberta

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-dataset-roberta%2F12479%3E%5C
**Category:** 🤗Datasets\
**Created:** [December 2, 2021, 6:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-dataset-roberta%2F12479 "2021-12-02T18:26:46Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![pere](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pere/32/40613_2.png) [@pere](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpere)%5C
**Post date:** [December 2, 2021, 6:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-dataset-roberta%2F12479%2F1 "2021-12-02T18:26:46Z")

</div>

Anyone know of RoBERTa pretraining script with support for Dataset streadmin?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [December 7, 2021, 12:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-dataset-roberta%2F12479%2F2 "2021-12-07T12:07:52Z")

</div>

Hi ! I don’t think the community has already shared a script for RoBERTa pretraining using dataset streaming yet. However if you’re interested in looking into this, here are a few pointers:

RoBERTa was trained with [BookCorpus](https://bookcorpus), [CC news](https://huggingface.co/datasets/cc_news) and [OpenWebText](https://openwebtext)

BookCorpus and OpenWebText have been replicated and open sourced as [BookCorpusOpen](https://huggingface.co/datasets/bookcorpusopen) and [OpenWebText2 (The Pile)](https://huggingface.co/datasets/the_pile_openwebtext2)

You can load and interleave the datasets with

```python
from datasets import load_dataset, interleave_datasets

def only_keep_text(example):
    return {"text": example["text"]}

bc = load_dataset("bookcorpusopen", split="train", streaming=True)
ccn = load_dataset("cc_news", split="train", streaming=True)
# this one currently has streaming issues - will fix soon
# owt = load_dataset("the_pile_openwebtext2", split="train", streaming=True)  

dataset = interleave_datasets([
    bc.map(only_keep_text),
    ccn.map(only_keep_text),
    # owt.map(only_keep_text)
])

```

Then you can check the documentation to see how to use it in a pytorch training loop: [Stream — datasets 1.16.1 documentation](https://huggingface.co/docs/datasets/stream.html#map)