# Token Classification as Pre-training task

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftoken-classification-as-pre-training-task%2F23393%3E%5C
**Category:** Models\
**Created:** [September 20, 2022, 12:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftoken-classification-as-pre-training-task%2F23393 "2022-09-20T12:39:52Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![initesh](https://avatars.discourse-cdn.com/v4/letter/i/54ee81/32.png) [@initesh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Finitesh)%5C
**Post date:** [September 20, 2022, 12:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftoken-classification-as-pre-training-task%2F23393%2F1 "2022-09-20T12:39:52Z")

</div>

Hi all,

I have a huge NER corpus (refer CrossNER paper). I want to pre-train BERT on this dataset and further fine-tune this NER-BERT instead of MLM+NSP BERT on custom NER datasets.

Any suggestions on how to do that?

Thanks,  
Nitesh