# Pre-training datasets for base and roberta

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-datasets-for-base-and-roberta%2F17853%3E%5C
**Category:** 🤗Datasets\
**Created:** [May 12, 2022, 3:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-datasets-for-base-and-roberta%2F17853 "2022-05-12T03:24:43Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![LinJiangxian](https://avatars.discourse-cdn.com/v4/letter/l/8e8cbc/32.png) [@LinJiangxian](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLinJiangxian)%5C
**Post date:** [May 12, 2022, 3:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-training-datasets-for-base-and-roberta%2F17853%2F1 "2022-05-12T03:24:43Z")

</div>

I find that the pre-training datasets for bert-base-uncased, roberta-base, roberta large… is same. wikipedia 19.3k and bookcorpus 8.4k. This is not consistent with the datasets described in paper. Is this an error? If it is, you may revise it.