# Gradient checkpointing + FSDP

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradient-checkpointing-fsdp%2F46930%3E%5C
**Category:** 🤗Accelerate\
**Created:** [July 15, 2023, 5:12pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradient-checkpointing-fsdp%2F46930 "2023-07-15T17:12:16Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![maxmatical](https://avatars.discourse-cdn.com/v4/letter/m/3ec8ea/32.png) [@maxmatical](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmaxmatical)%5C
**Post date:** [July 15, 2023, 5:12pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradient-checkpointing-fsdp%2F46930%2F1 "2023-07-15T17:12:16Z")

</div>

I wasn’t able to find any documentation on this, but if I want to use gradient checkpointing with FSDP training (assuming the `model.supports_gradient_checkpointing` is `True`), do i need to manually apply the wrapping like so

method 1

```auto
model: PretrainedModel = ...

check_fn = lambda submodule: isinstance(submodule, ...)

non_reentrant_wrapper = partial(
    checkpoint_wrapper,
    offload_to_cpu=False,
    checkpoint_impl=CheckpointImpl.NO_REENTRANT,
)

apply_activation_checkpointing(
    model, checkpoint_wrapper_fn=non_reentrant_wrapper, check_fn=check_fn
)

model = accelerator.prepare(model) 
...

```

or does `model.gradient_checkpointing_enable` take care of that, so instead

method 2

```auto
model: PretrainedModel = ...
model.gradient_checkpointing_enable()
model = accelerator.prepare(model) 
...

```

which is the correct method? And in the case that the model doesn’t not support gradient checkpointing, can i manually apply gradient checkpointing via method 1?

Another question:  
The gradient checkpointing and FSDP wrapping policy should to apply to the same layers, which is fine if using the `TRANSFORMER_BASED_WRAP` policy, but what about using `SIZE_BASED_WRAP` or `NO_WRAP`?

Or is my understanding incorrect, and checkpointing logic and FSDP wrapping policy can be independent of one another

---

<div class="post-metadata">

**Author:** ![jamesagilesoda](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jamesagilesoda/32/16356_2.png) [@jamesagilesoda](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjamesagilesoda)%5C
**Post date:** [August 22, 2023, 10:48am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgradient-checkpointing-fsdp%2F46930%2F2 "2023-08-22T10:48:05Z")

</div>

I applied the method 2:

```auto
model: PretrainedModel = ...
model.gradient_checkpointing_enable()
model = accelerator.prepare(model)

```