# Accelerate FSDP config prompts
**URL:**
**Author:**  [@ablam](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fablam)%5C
**Post date:** [August 5, 2022, 3:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerate-fsdp-config-prompts%2F21262%2F1 "2022-08-05T03:15:09Z")
Hello. I’m using accelerate to leverage FSDP as per the tutorial [here](https://huggingface.co/blog/pytorch-fsdp). I would like to finetune CodeBert using [run\_mlm\_no\_trainer.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm_no_trainer.py) on a custom dataset. I’ve run `accelerate config` but I’m not sure what the prompt → `transformer_layer_cls_to_wrap` entails - be it for my model or even in the examples provided.
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [August 5, 2022, 1:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerate-fsdp-config-prompts%2F21262%2F2 "2022-08-05T13:10:02Z")
cc @smangrul
---
**Author:**  [@smangrul](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsmangrul)%5C
**Post date:** [August 5, 2022, 3:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerate-fsdp-config-prompts%2F21262%2F3 "2022-08-05T15:27:21Z")
Hello @ablam, the blog post is outdated as the FSDP features have been upgraded in PyTorch version 1.12.0. As such, all these new features have been integrated into HF Accelerate. For transformer-based models, PyTorch teams suggested using the `transformer_auto_wrap` policy. In this policy, the user has to specify the case-sensitive name of an encoder/decoder block comprising of the Multi-Head Attention layer followed by the Feedforward layer. For example, in the T5 model, `T5Block` is the name for the attention block used by the model for N such layers/blocks in the encoder and decoder. Similarly, for the BERT model, it is `BertLayer` and for GPT2 it is `GPT2Block`. Below is an example of the accelerate config for the `bert-base-cased` model :
```auto
compute_environment: LOCAL_MACHINE
deepspeed_config: {}
distributed_type: FSDP
downcast_bf16: 'no'
fsdp_config:
fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
fsdp_backward_prefetch_policy: BACKWARD_PRE
fsdp_offload_params: false
fsdp_sharding_strategy: 1
fsdp_state_dict_type: FULL_STATE_DICT
fsdp_transformer_layer_cls_to_wrap: BertLayer
machine_rank: 0
main_process_ip: null
main_process_port: null
main_training_function: main
mixed_precision: fp16
num_machines: 1
num_processes: 2
use_cpu: false
```
You will have to print the model and check the name for the attention block and pass it as the value for `transformer_layer_cls_to_wrap`. I hope this helps.
---
**Author:**  [@ablam](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fablam)%5C
**Post date:** [August 9, 2022, 8:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerate-fsdp-config-prompts%2F21262%2F4 "2022-08-09T20:26:39Z")
Yes, it does. Found the corresponding reference in my model. Thank you!
---
**Author:**  [@Colorful](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FColorful)%5C
**Post date:** [July 16, 2023, 4:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerate-fsdp-config-prompts%2F21262%2F5 "2023-07-16T16:14:30Z")
Hi,
I have a question about `fsdp_transformer_layer_cls_to_wrap`: How can I know the transformer layer for different models? If I want to accelerate LLaMa, can I directly set this parameter to `LlamaLayer`?
---
**Author:**  [@smangrul](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsmangrul)%5C
**Post date:** [September 15, 2023, 9:59am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Faccelerate-fsdp-config-prompts%2F21262%2F6 "2023-09-15T09:59:56Z")
Hello @Colorful, the docs have been update here recently [Fully Sharded Data Parallel (huggingface.co)](https://huggingface.co/docs/accelerate/main/en/usage_guides/fsdp) answering your question. Specifically, the highlighted part below.
