# Custom Tasks and BERT Fine Tuning
**URL:**
**Author:**  [@npan1990](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnpan1990)%5C
**Post date:** [August 26, 2020, 11:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-tasks-and-bert-fine-tuning%2F874%2F1 "2020-08-26T11:47:56Z")
I am using the transformers library to get embeddings for sentences and tokens. More specifically I use the first token embedding [CLS] for the embedding that represents the sentence and I compare sentences using cosine similarity. This approach is naive and completely unsupervised.
Now I would like to gain some experience in fine tuning the model: For example how to fine tune BERT for NER and how to use BERT for sentence pairs. How can I study how to fine tune BERT?
---
**Author:**  [@rgwatwormhill](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frgwatwormhill)%5C
**Post date:** [August 26, 2020, 11:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-tasks-and-bert-fine-tuning%2F874%2F2 "2020-08-26T23:03:29Z")
nice tutorial on tuning using transformers with pytorch by Chris McCormick here [https://mccormickml.com/2019/07/22/BERT-fine-tuning/#4-train-our-classification-model](https://mccormickml.com/2019/07/22/BERT-fine-tuning/#4-train-our-classification-model). See also his youtube videos and other posts.
If you want to make a custom model, try this by abhimishra [https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers\_multi\_label\_classification.ipynb](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_multi_label_classification.ipynb)
---
**Author:**  [@avinashpaul](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Favinashpaul)%5C
**Post date:** [October 30, 2020, 6:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-tasks-and-bert-fine-tuning%2F874%2F3 "2020-10-30T06:27:54Z")
what is the difference between fine tune and custom model ?
---
**Author:**  [@rgwatwormhill](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frgwatwormhill)%5C
**Post date:** [October 30, 2020, 1:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-tasks-and-bert-fine-tuning%2F874%2F4 "2020-10-30T13:59:09Z")
The standard BERT-base has 12 layers each of 12 heads, using 768 dimensions for the vector encoding. Those values cannot be changed after the model has been created and pre-trained. When you fine-tune a BERT-base model to solve your own task, you change the values of the weight parameters within the model, but you don’t change the number or layers or heads, and you don’t change the 768 dimensions.
If you create and pre-train your own BERT model, then you can choose any number of layers or heads or dimensions, and you can choose how many tokens your BERT has in its vocabulary. I believe we would call this a custom model.
By the way, there are several different sizes of BERT model available, pre-trained. For example, see this link for BERT-small, BERT-tiny etc, which were created by the original BERT team Devlin et al
> **[GitHub - google-research/bert: TensorFlow code and pre-trained models for BERT](https://github.com/google-research/bert)**
>
> TensorFlow code and pre-trained models for BERT. Contribute to google-research/bert development by creating an account on GitHub.
You might also be interested in BERT-like models that have different sizes and have also been pre-trained in different ways, such as DistilBERT, ALBERT and RoBERTa
---
**Author:**  [@rgwatwormhill](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frgwatwormhill)%5C
**Post date:** [October 30, 2020, 2:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-tasks-and-bert-fine-tuning%2F874%2F5 "2020-10-30T14:03:33Z")
Hello again, I’ve just re-read what I wrote above, and realised the second sentence is not quite right. The values _can_ be changed after pre-training. It is called **pruning**. However, those values are not altered during normal fine-tuning.