# Keep NSP head after BertForPretraining

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkeep-nsp-head-after-bertforpretraining%2F14307%3E%5C
**Category:** 🤗Transformers\
**Created:** [February 1, 2022, 6:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkeep-nsp-head-after-bertforpretraining%2F14307 "2022-02-01T18:14:31Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![tzuhsial](https://avatars.discourse-cdn.com/v4/letter/t/b5a626/32.png) [@tzuhsial](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftzuhsial)%5C
**Post date:** [February 1, 2022, 6:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkeep-nsp-head-after-bertforpretraining%2F14307%2F1 "2022-02-01T18:14:31Z")

</div>

I am pretraining with MLM + NSP (`BertForPretraining`), and finetuning with NSP (`BertForNextSentencePrediction`).

Is there an elegant way which I can keep the NSP head from the pretrained model?

Thanks!

---

<div class="post-metadata">

**Author:** ![tzuhsial](https://avatars.discourse-cdn.com/v4/letter/t/b5a626/32.png) [@tzuhsial](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftzuhsial)%5C
**Post date:** [February 1, 2022, 6:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkeep-nsp-head-after-bertforpretraining%2F14307%2F2 "2022-02-01T18:30:58Z")

</div>

Validated that `using BertForNextSentencePrediction` does indeed keep the weights of NSP head of `BertForPreTraining`