# Fine-tuning TrOCR on custom dataset

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-trocr-on-custom-dataset%2F39108%3E%5C
**Category:** 🤗Transformers\
**Created:** [May 8, 2023, 2:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-trocr-on-custom-dataset%2F39108 "2023-05-08T14:57:17Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![FredrikPD](https://avatars.discourse-cdn.com/v4/letter/f/5f8ce5/32.png) [@FredrikPD](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FFredrikPD)%5C
**Post date:** [May 8, 2023, 2:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-trocr-on-custom-dataset%2F39108%2F1 "2023-05-08T14:57:17Z")

</div>

Hello, I am building an OCR system where I use the TrOCR for the text recognition as shown below:

```auto
processor = TrOCRProcessor.from_pretrained("microsoft/trocr-base-printed")
model = VisionEncoderDecoderModel.from_pretrained("microsoft/trocr-base-printed")
optimizer = optim.AdamW(model.parameters(), lr=5e-5)

```

Since we are to use the OCR system for receipts we chose the “printed” pretrained model. We use a dataset of 5000 bounding boxes where each contain a word. However we experience that all metrics (cer, precision) and loss increase for each epoch we run. We cant figure out why the model performs more poorly for each epoch. This is how we train:

```auto
for epoch in range(self.epochs):
            self.model.train()
            train_loss = 0.0
            for batch in tqdm(self.train_dataloader):
                for k, v in batch.items():
                    batch[k] = v.to(self.device)
                outputs = self.model(**batch)
                loss = outputs.loss
                loss.backward()
                self.optimizer.step()
                self.optimizer.zero_grad()
                train_loss += loss.item()

```

Does anyone have an idea to what we might be doing wrong?

---

<div class="post-metadata">

**Author:** ![rajsabi](https://avatars.discourse-cdn.com/v4/letter/r/e95f7d/32.png) [@rajsabi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frajsabi)%5C
**Post date:** [October 18, 2023, 11:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-trocr-on-custom-dataset%2F39108%2F2 "2023-10-18T11:09:06Z")

</div>

This is performing poor because you are using already fined-tuned model for training. this microsoft/trocr-base-printed is fine-tuned on SROIE dataset. You should choose some pre-trained model of trocr like **microsoft/trocr-base-stage1**. select from here [Models - Hugging Face](https://huggingface.co/models?sort=trending&search=trocr).