# Very poor model performance post-training

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fvery-poor-model-performance-post-training%2F11252%3E%5C
**Category:** 🤗Transformers\
**Created:** [November 1, 2021, 12:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fvery-poor-model-performance-post-training%2F11252 "2021-11-01T12:36:16Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![sivakhno](https://avatars.discourse-cdn.com/v4/letter/s/e274bd/32.png) [@sivakhno](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsivakhno)%5C
**Post date:** [November 1, 2021, 12:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fvery-poor-model-performance-post-training%2F11252%2F1 "2021-11-01T12:36:17Z")

</div>

Helo All - I have created a flavour of Bert model with new embeddings by reusing existing Bert modules. My workflow is as follows:

1. Modify run\_mlm.py to add new embeddings in forward loop
2. Run MLM training and save pre-trained model.
3. Modify BertModel for classification (apporx 40 classes) by adapting embedding layer and forward.
4. Load pre-trained model from step 2, fine-tune on classification task and save.
5. Load for classification prediction.

The problem is that in step (4) I see exceptional performance on validation set. But when I load a model to make predictions (step 5) it performs extremely poorly and consistently selects only few classes. I made sure that the same code is run in validation loop as in prediction. I can also see saved new embedding layers when loading with pytorch load. Off-the shelf Bert model works well on the dataset, suggesting that the issue is not with DataLoader.

I am stack as I am not sure how best to debug this next - any suggestions would be most appreciated!  
Many thanks for your help in advance!