# ByT5 for text classification

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbyt5-for-text-classification%2F8831%3E%5C
**Category:** Models\
**Created:** [July 30, 2021, 10:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbyt5-for-text-classification%2F8831 "2021-07-30T10:09:02Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![janyfe](https://avatars.discourse-cdn.com/v4/letter/j/ed655f/32.png) [@janyfe](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjanyfe)%5C
**Post date:** [July 30, 2021, 10:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbyt5-for-text-classification%2F8831%2F1 "2021-07-30T10:09:02Z")

</div>

Hi!

Facing a problem whilst trying to finetune the ByT5 model for the text classification task.

I’ve tried to use this notebook [exploring-T5/t5\_fine\_tuning.ipynb at master · patil-suraj/exploring-T5 · GitHub](https://github.com/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb) by @valhalla for ByT5 finetuning on the text classification task. When I start from ‘google/byt5-small’ I get really strange results. A model always generates negative sentiment label (‘n’ in my code)

 ![Screenshot 2021-07-30 at 13.04.30](https://us1.discourse-cdn.com/hellohellohello/original/2X/9/90f50f9add3583cab9c7eece4572b312fff1e70b.png)

When I switch to t5-small pretrained checkpoint the results are reasonable

 ![Screenshot 2021-07-30 at 12.19.55](https://us1.discourse-cdn.com/hellohellohello/original/2X/1/19282c23adc671385c1e3e2f504f60d8d699f585.png)

What am I missing here? Any piece of advice would be much appreciated!

---

<div class="post-metadata">

**Author:** ![Chandrai](https://avatars.discourse-cdn.com/v4/letter/c/8baadc/32.png) [@Chandrai](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FChandrai)%5C
**Post date:** [June 20, 2022, 5:37am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbyt5-for-text-classification%2F8831%2F2 "2022-06-20T05:37:38Z")

</div>

Hello @janyfe . Could you find a solution for this? I am also facing same issue with multi-class text classification task. Among 6 classes, byt5 always predicts 1 class.

---

<div class="post-metadata">

**Author:** ![janyfe](https://avatars.discourse-cdn.com/v4/letter/j/ed655f/32.png) [@janyfe](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjanyfe)%5C
**Post date:** [June 20, 2022, 11:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbyt5-for-text-classification%2F8831%2F3 "2022-06-20T11:56:32Z")

</div>

Hello @Chandrai! No, unfortunately, I wasn’t able to find the solution and gave up using byt5 model.

---

<div class="post-metadata">

**Author:** ![mohanadhafez](https://avatars.discourse-cdn.com/v4/letter/m/0ea827/32.png) [@mohanadhafez](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmohanadhafez)%5C
**Post date:** [July 21, 2024, 7:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbyt5-for-text-classification%2F8831%2F4 "2024-07-21T07:27:55Z")

</div>

Hello @Chandrai. Did you find a solution?