# Question about Gradient Accumulation step in Trainer
**URL:**
**Author:**  [@ben9004](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fben9004)%5C
**Post date:** [September 10, 2021, 8:38am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestion-about-gradient-accumulation-step-in-trainer%2F9876%2F1 "2021-09-10T08:38:28Z")
I can see that gradient accumulation step helps to increase batch size.
and also I can understand if the model has Batch Norm layer, gradient accumulation will not guarantee the exact same performance as the model that we trained in a large batch size (not using accumulation)
but, most of models in Transformers are based on transformer architecture which utilizes layer normalization,
so does that mean Can I guarantee that the trained model would give same metric performance in both ways? (e.g batch size 64 with 4 batch per device \* 4 gpus \* 4 accumulation step == batch size 64 with 16 batch per device \* 4 gpus )
In short,
My question is for transformer models which use layer normalization, will give same model performance between train batch size in once and using gradient accumulation steps
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [September 10, 2021, 1:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestion-about-gradient-accumulation-step-in-trainer%2F9876%2F2 "2021-09-10T13:18:52Z")
Yes, layer normalization does track statistics, so you will get the exact same thing with 4 batch size \* 4 gradient accumulation or 16 batch size (and that would not be the case with neural nets using BatchNorm indeed).
---
**Author:**  [@ben9004](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fben9004)%5C
**Post date:** [September 10, 2021, 2:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fquestion-about-gradient-accumulation-step-in-trainer%2F9876%2F3 "2021-09-10T14:50:51Z")
Thanks! your reply has been so much help to me