# How to create the fsdp\_config json file for Trainer?
**URL:**
**Author:**  [@narai](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnarai)%5C
**Post date:** [May 31, 2023, 6:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-the-fsdp-config-json-file-for-trainer%2F41685%2F1 "2023-05-31T18:56:09Z")
I want to do Alpaca style training ([GitHub - tatsu-lab/stanford\_alpaca: Code and documentation to train Stanford's Alpaca models, and generate the data.](https://github.com/tatsu-lab/stanford_alpaca/tree/main) , [stanford\_alpaca/train.py at main · tatsu-lab/stanford\_alpaca · GitHub](https://github.com/tatsu-lab/stanford_alpaca/blob/main/train.py#L221)) with FSDP, with model sharding. But I am deviating from the alpaca training method in several ways: I want to launch with:
python3 -m torch.distributed.launch --nproc\_per\_node=4 train-pythia-script.py
(which as far as I can tell, is the best accepted way to use Trainer with FSDP in May 2023).
Also, I am directly specifying arguments in the Trainer init:
training\_args = TrainingArguments(
output\_dir=“./results”,
fp16=True,
overwrite\_output\_dir=True,
num\_train\_epochs=1,
per\_device\_train\_batch\_size=1,
per\_device\_eval\_batch\_size=1,
gradient\_accumulation\_steps=8, # alpaca
evaluation\_strategy=“no”, # alpaca
save\_steps=10\_000,
save\_total\_limit=1, # alpaca
learning\_rate=2e-5, # alpaca
weight\_decay=0.0, # alpaca
warmup\_ratio=0.03, # alpaca
lr\_scheduler\_type=‘cosine’, # alpaca
logging\_steps=1, # alpaca
fsdp=“full\_shard auto\_wrap”, # alpaca
fsdp\_config=‘fsdp\_config\_pythia.json’
)
I am using the MAIN branch of transformers.
When I tried to include fsdp\_transformer\_layer\_cls\_to\_wrap argument in Trainer, the code gave me a FutureWarning,
/home/ubuntu/project/transformers/src/transformers/training\_args.py:1462: FutureWarning: using `--fsdp_transformer_layer_cls_to_wrap` is deprecated. Use fsdp\_config instead
and an error:
Exception : raise Exception(“Could not find the transformer layer class to wrap in the model.”)Could not find the transformer layer class to wrap in the model.
so I changed to the argument fsdp\_config=‘fsdp\_config\_pythia.json’
How do I create the fsdp\_config json file?
---
**Author:**  [@narai](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnarai)%5C
**Post date:** [May 31, 2023, 7:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-the-fsdp-config-json-file-for-trainer%2F41685%2F2 "2023-05-31T19:32:12Z")
I think it is working now, after
1. downgrading to transformers 4.26.1 (which does not use the fsdp\_config argument)
2. removing fsdp\_config argument
3. adding back the fsdp\_transformer\_layer\_cls\_to\_wrap argument
It is using less memory compared to non-FSDP mode, so I think the model is actually being sharded.
---
**Author:**  [@narai](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnarai)%5C
**Post date:** [May 31, 2023, 8:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-the-fsdp-config-json-file-for-trainer%2F41685%2F3 "2023-05-31T20:02:10Z")
Unfortunately this ‘fix’ worked for my Pythia model test, but I need the newer version of transformers for Llama model support …
---
**Author:**  [@muellerzr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmuellerzr)%5C
**Post date:** [June 5, 2023, 1:12am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-the-fsdp-config-json-file-for-trainer%2F41685%2F4 "2023-06-05T01:12:16Z")
cc @smangrul
---
**Author:**  [@lucasjin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flucasjin)%5C
**Post date:** [June 19, 2023, 3:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-create-the-fsdp-config-json-file-for-trainer%2F41685%2F5 "2023-06-19T15:03:12Z")
Same here, it would be better to post a link (to doc or something) indicates users how to add fsconfig
FutureWarning: using `--fsdp_transformer_layer_cls_to_wrap` is deprecated. Use fsdp\_config instead
rather than just tell this is deprecated. An intuitive example can help reduce waste users time .