# Gradient Checkpointing with FSDP efficiency

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradient-checkpointing-with-fsdp-efficiency%2F51386%3E%5C
**Category:** 🤗Transformers\
**Created:** [August 20, 2023, 5:08am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradient-checkpointing-with-fsdp-efficiency%2F51386 "2023-08-20T05:08:46Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![StringChaos](https://avatars.discourse-cdn.com/v4/letter/s/43a26b/32.png) [@StringChaos](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FStringChaos)%5C
**Post date:** [August 20, 2023, 5:08am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradient-checkpointing-with-fsdp-efficiency%2F51386%2F1 "2023-08-20T05:08:46Z")

</div>

I have been fine-tuning a llama-style model on 8 40GB A100 GPUs with flash attention and FSDP. I was trying training argument combinations and found that turning off gradient checkpointing actually slowed the training throughput which was very surprising. I was curious about why this might happen since gradient checkpointing requires re-computations! (I suspect this might be caused by FSDP since for smaller models turning gradient checkpointing off improves throughput)