# Fine tuning bert on next sentence prediction task
**URL:**
**Author:**  [@zenonas009](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzenonas009)%5C
**Post date:** [September 13, 2020, 12:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-bert-on-next-sentence-prediction-task%2F1101%2F1 "2020-09-13T12:15:18Z")
I am trying to fine-tune Bert using the Huggingface library on next sentence prediction task. I looked at the tutorial and I am trying to use `DataCollatorForNextSentencePrediction` and `TextDatasetForNextSentencePrediction` . When I am using that I get the following error(use the pastebin link to see the error)[https://pastebin.pl/view/bde2c3d4](https://pastebin.pl/view/bde2c3d4). I have provided my code bellow.
============Code================
def train(bert\_model,bert\_tokenizer,path,eval\_path=None):
out\_dir = “/content/drive/My Drive/next\_sentence/”
training\_args = TrainingArguments(
output\_dir=out\_dir,
overwrite\_output\_dir=True,
num\_train\_epochs=1,
per\_device\_train\_batch\_size=30,
save\_steps=10000,
save\_total\_limit=2,
)
data\_collator = DataCollatorForNextSentencePrediction(
tokenizer=bert\_tokenizer,mlm=False,block\_size=512,nsp\_probability =0.5
)
```
dataset = TextDatasetForNextSentencePrediction(
tokenizer = bert_tokenizer,
file_path=path,
block_size=512,
)
trainer = Trainer(
model=bert_model,
args=training_args,
train_dataset=dataset,
data_collator=data_collator,
)
trainer.train()
trainer.save_model(out_dir)
def main():
print("Running main")
bert_tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
bert_model = BertForNextSentencePrediction.from_pretrained("bert-base-cased")
train_data_set_path = "/content/drive/My Drive/next_sentence/line_data_set_file.txt"
train(bert_model,bert_tokenizer,train_data_set_path)
#prepare_data_set(bert_tokenizer)
main()
```
---
**Author:**  [@nbroad](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnbroad)%5C
**Post date:** [September 14, 2020, 2:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-bert-on-next-sentence-prediction-task%2F1101%2F2 "2020-09-14T02:13:31Z")
Can you fix the formatting in your post? It would make it easier to read 🙂
---
**Author:**  [@zenonas009](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzenonas009)%5C
**Post date:** [September 14, 2020, 8:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-bert-on-next-sentence-prediction-task%2F1101%2F3 "2020-09-14T08:13:39Z")
I added the error log to a pastebin link. Might be easier to read it that way 😃
---
**Author:**  [@vblagoje](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvblagoje)%5C
**Post date:** [September 29, 2020, 12:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-bert-on-next-sentence-prediction-task%2F1101%2F4 "2020-09-29T12:27:22Z")
Hey @zenonas009 did you make any progress on the training?
---
**Author:**  [@zenonas009](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzenonas009)%5C
**Post date:** [September 29, 2020, 12:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-bert-on-next-sentence-prediction-task%2F1101%2F5 "2020-09-29T12:47:11Z")
Hello @vblagoje. Yes , I found a couple of problems and managed to fix them.
1. I was not creating the text file in a correct way so the dataset class couldn’t parse it and create the examples. That was causing the error I have on the pastebin link.
2. The datacollator class seems to return “masked\_lm\_labels” which I don’t know why and what is the relevance when doing nsp. I removed that key and everything seems to be working fine.
3. Also I noticed that datacollator will return “next\_sentence\_label” instead of “labels”.That seemed a problem for the Trainer class so I changed “next\_sentence\_label” into “labels”.
I just saw the source code, copy it locally and made the changes locally.
I am currently getting the progress bars when training starts and I am waiting for it to finish training.
---
**Author:**  [@vblagoje](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvblagoje)%5C
**Post date:** [September 30, 2020, 12:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-bert-on-next-sentence-prediction-task%2F1101%2F6 "2020-09-30T12:08:08Z")
How did you prepare 17Gb dataset and then continually feed it to the model? That’s where I got stuck.