# Why is use\_cache incompatible with gradient checkpointing?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-use-cache-incompatible-with-gradient-checkpointing%2F18811%3E%5C
**Category:** 🤗Transformers\
**Created:** [June 7, 2022, 10:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-use-cache-incompatible-with-gradient-checkpointing%2F18811 "2022-06-07T10:25:46Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ratishsp](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@ratishsp](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fratishsp)%5C
**Post date:** [June 7, 2022, 10:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-use-cache-incompatible-with-gradient-checkpointing%2F18811%2F1 "2022-06-07T10:25:46Z")

</div>

Hi,  
I see the below snippet in modeling\_t5.py. I wanted to understand why use\_cache is incompatible with gradient checkpointing.

> <https://github.com/huggingface/transformers/blob/9aa230aa2fb17e4a7da4c61fccaddafe410f7ed1/src/transformers/models/t5/modeling_t5.py#L1008-L1011>

Thanks.

---

<div class="post-metadata">

**Author:** ![lianghsun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lianghsun/32/57774_2.png) [@lianghsun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flianghsun)%5C
**Post date:** [June 8, 2022, 5:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-use-cache-incompatible-with-gradient-checkpointing%2F18811%2F2 "2022-06-08T05:46:28Z")

</div>

Hi, there. I face the same problem in `run_clm.py` when I set `--gradient_checkpointing true`. However I do not find any config that I can set on `run_clm.py`, does anyone know?

---

<div class="post-metadata">

**Author:** ![luisarmando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/luisarmando/32/24205_2.png) [@luisarmando](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fluisarmando)%5C
**Post date:** [October 27, 2022, 9:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-use-cache-incompatible-with-gradient-checkpointing%2F18811%2F3 "2022-10-27T21:11:09Z")

</div>

> [@lianghsun](#):
>
> –gradient\_checkpointing true

Anybody knows how to fix this?

---

<div class="post-metadata">

**Author:** ![rs2992](https://avatars.discourse-cdn.com/v4/letter/r/6bbea6/32.png) [@rs2992](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frs2992)%5C
**Post date:** [March 24, 2023, 4:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-use-cache-incompatible-with-gradient-checkpointing%2F18811%2F4 "2023-03-24T16:33:03Z")

</div>

Honestly, I’ve just ignored it. It automatically disables the use\_cache function. I’m about to remove the warning altogether.

`use_cache=True` is incompatible with gradient checkpointing. **Setting `use_cache=False`**

---

<div class="post-metadata">

**Author:** ![kira](https://avatars.discourse-cdn.com/v4/letter/k/c4cdca/32.png) [@kira](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkira)%5C
**Post date:** [July 7, 2023, 12:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-use-cache-incompatible-with-gradient-checkpointing%2F18811%2F5 "2023-07-07T12:44:16Z")

</div>

its related to `past_key_values`, you can disable this warning by setting  
`model.config.use_cache = False`, when using gradient checkpointing during training.  
but during inference make sure to set it back to `True`.

---

<div class="post-metadata">

**Author:** ![mmiakashs](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mmiakashs/32/1959_2.png) [@mmiakashs](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmmiakashs)%5C
**Post date:** [August 25, 2023, 9:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-use-cache-incompatible-with-gradient-checkpointing%2F18811%2F6 "2023-08-25T21:32:46Z")

</div>

Could you please share what are the purposes of `use_cache`? Thanks.

---

<div class="post-metadata">

**Author:** ![kira](https://avatars.discourse-cdn.com/v4/letter/k/c4cdca/32.png) [@kira](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkira)%5C
**Post date:** [September 15, 2023, 1:39am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-use-cache-incompatible-with-gradient-checkpointing%2F18811%2F7 "2023-09-15T01:39:01Z")

</div>

> **[Generation](https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.GenerationConfig.use_cache)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.