# Gradient\_checkpointing = True results in error

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradient-checkpointing-true-results-in-error%2F10744%3E%5C
**Category:** 🤗Transformers\
**Created:** [October 13, 2021, 4:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradient-checkpointing-true-results-in-error%2F10744 "2021-10-13T16:32:24Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ArnauC](https://avatars.discourse-cdn.com/v4/letter/a/2acd7d/32.png) [@ArnauC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FArnauC)%5C
**Post date:** [October 13, 2021, 4:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradient-checkpointing-true-results-in-error%2F10744%2F1 "2021-10-13T16:32:24Z")

</div>

Hi all,

I’m trying to finetune a summarization model (bigbird-pegasus-large-bigpatent) on my own data.  
Of course even with premium colab I’m having memory issues, so I tried to set gradient\_checkpointing = True in the Seq2SeqTrainingArguments, which is supposed to save some memory altgough increasing the computation time.

The problem is that when starting the training this argument rises an error:

AttributeError: module ‘torch.utils’ has no attribute ‘checkpoint’

Has anyone experienced this same error?  
I read in the Github discussion:  
[https://github.com/huggingface/transformers/issues/9617](https://github.com/huggingface/transformers/issues/9617)  
[https://github.com/huggingface/transformers/issues/11193](https://github.com/huggingface/transformers/issues/11193)  
[https://github.com/huggingface/transformers/issues/9919](https://github.com/huggingface/transformers/issues/9919)

that in some other cases the same error was appearing but it was supposed to be solved here:  
[https://github.com/huggingface/transformers/pull/9626](https://github.com/huggingface/transformers/pull/9626)

Any help would be appreciated.  
Thanks

---

<div class="post-metadata">

**Author:** ![sfalk](https://avatars.discourse-cdn.com/v4/letter/s/5e9695/32.png) [@sfalk](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsfalk)%5C
**Post date:** [December 16, 2021, 2:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradient-checkpointing-true-results-in-error%2F10744%2F2 "2021-12-16T14:16:06Z")

</div>

Hi! I am facing a similar issue. Have you been able to solve it?

The code that causing the problem here is the following:

```auto
model_path = "facebook/s2t-small-librispeech-asr"

# Initialize the model
model = Speech2TextForConditionalGeneration.from_pretrained(model_path)

model = model.eval()
# Attach decoder
model = SpeechRecognizer(model, labels=labels)

# Apply quantization / script / optimize for mobile
quantized_model = torch.quantization.quantize_dynamic(model, qconfig_spec={torch.nn.Linear}, dtype=torch.qint8)
scripted_model = torch.jit.script(quantized_model)

```

The `SpeechRecognizer` is just a simple `torch.nn.Module` wrapper.

---

<div class="post-metadata">

**Author:** ![ArnauC](https://avatars.discourse-cdn.com/v4/letter/a/2acd7d/32.png) [@ArnauC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FArnauC)%5C
**Post date:** [December 17, 2021, 11:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradient-checkpointing-true-results-in-error%2F10744%2F3 "2021-12-17T11:13:15Z")

</div>

Hi! I think that instead of adding gradient\_checkpointing as an argument to training arguments, I used this line when I defined the model:

model = AutoModelForSeq2SeqLM.from\_pretrained(model\_checkpoint)  
model.gradient\_checkpointing\_enable()

Anyway we ended up training this model in a GCP, it was too big.

Hope this helps!

---

<div class="post-metadata">

**Author:** ![jaideepcs](https://avatars.discourse-cdn.com/v4/letter/j/ecae2f/32.png) [@jaideepcs](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjaideepcs)%5C
**Post date:** [February 22, 2023, 1:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradient-checkpointing-true-results-in-error%2F10744%2F4 "2023-02-22T13:36:28Z")

</div>

@ArnauC did it behave wierdly ?