# pre-train\_BERT for a specific corpus

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-bert-for-a-specific-corpus%2F84861%3E%5C
**Category:** 🤗Transformers\
**Created:** [May 2, 2024, 3:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-bert-for-a-specific-corpus%2F84861 "2024-05-02T15:01:02Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jaafer](https://avatars.discourse-cdn.com/v4/letter/j/9d8465/32.png) [@Jaafer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJaafer)%5C
**Post date:** [May 2, 2024, 3:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpre-train-bert-for-a-specific-corpus%2F84861%2F1 "2024-05-02T15:01:02Z")

</div>

**pre-train bert**  
HI  
i am trying to pre\_train bert in my own corpus  
but got this errors

```auto
!pip install accelerate -U
!pip install transformers
import torchvision, torchaudio
from accelerate import Accelerator
from transformers import DataCollatorForLanguageModeling,TextDataset,TrainingArguments

from transformers import BertTokenizer, BertForPreTraining

tokenizer = BertTokenizer.from_pretrained('Jaafer/code-search-net-tokenizer')

model = BertForPreTraining.from_pretrained('bert-base-uncased')
!pip install accelerate -U
!pip install transformers
import torchvision, torchaudio
from accelerate import Accelerator
from transformers import DataCollatorForLanguageModeling,TextDataset,TrainingArguments

from transformers import BertTokenizer, BertForPreTraining

tokenizer = BertTokenizer.from_pretrained('Jaafer/code-search-net-tokenizer')

model = BertForPreTraining.from_pretrained('bert-base-uncased')
%%time
data="/kaggle/input/ontology/ontology.txt"

from transformers import LineByLineTextDataset

dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path=data,
    block_size=128,
)
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=True, mlm_probability=0.15)
model.to(device)
from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./EsperBERTo",
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_gpu_train_batch_size=64,
    save_steps=10_000,
    save_total_limit=2,
    prediction_loss_only=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
)
trainer.train()

```

 ![er](https://us1.discourse-cdn.com/hellohellohello/original/3X/2/7/27540c083687bfc486d41692d7be172e6971c7c6.png)