# Questions on distilling \[from\] T5

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestions-on-distilling-from-t5%2F1193%3E%5C
**Category:** 🤗Transformers\
**Created:** [September 19, 2020, 12:43am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestions-on-distilling-from-t5%2F1193 "2020-09-19T00:43:12Z")\
**Posts on this page:** 1\
**Showing post:** 7

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsshleifer)%5C
**Post date:** [September 19, 2020, 5:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestions-on-distilling-from-t5%2F1193%2F7 "2020-09-19T17:28:53Z")

</div>

Exactly.  
Complicated version of that:

> [@Diverse Generations for pseudolabeling](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdiverse-generations-for-pseudolabeling%2F1189%2F6)%3A
>
> I read the FAIR [self-training paper](https://arxiv.org/pdf/1909.13788.pdf) and it is very relevant, but I am struggling to understand the specifics of the approach. Things I understood: “use beam search decoding (beam size 5) to create the pseudo targets and to report BLEU on test set.” This outperforms sampling. They train with dropout rate at 0.3. This helps. They don’t do any cleaning or checking of the pseudolabels against the ground truth target. all experiments are run on 8 GPUs with an effective batch size of 33K tokens. Th…

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestions-on-distilling-from-t5%2F1193)._