# Easy way to implement annealing temperature softmax

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Feasy-way-to-implement-annealing-temperature-softmax%2F2966%3E%5C
**Category:** 🤗Transformers\
**Created:** [December 26, 2020, 12:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Feasy-way-to-implement-annealing-temperature-softmax%2F2966 "2020-12-26T00:17:32Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![zeyuyun1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/zeyuyun1/32/524_2.png) [@zeyuyun1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzeyuyun1)%5C
**Post date:** [December 26, 2020, 12:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Feasy-way-to-implement-annealing-temperature-softmax%2F2966%2F1 "2020-12-26T00:17:32Z")

</div>

Hi friends,

I wonder if there’s a easy way to implement annealing temperature softmax:

For example, I want to change [314](https://github.com/huggingface/transformers/blob/61443cd7d917ef323a799ee27bb4abc4344f0d11/src/transformers/models/bert/modeling_bert.py#L314) in modeling\_bert.py from

`attention_probs = nn.Softmax(dim=-1)(attention_scores)`

to

`attention_probs = nn.Softmax(dim=-1)(attention_scores/temp)`

where “temp” is a variable decaying as the training process goes on according to a scheduler like the learning rate.

Thank you!

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [January 4, 2021, 1:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Feasy-way-to-implement-annealing-temperature-softmax%2F2966%2F2 "2021-01-04T13:01:01Z")

</div>

You can just do that change 🙂  
Model files are kept completely independent from each other just so you can easily tweak them for experiments like this.