# Parameters that contribute to GPU Memory

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparameters-that-contribute-to-gpu-memory%2F63213%3E%5C
**Category:** Models\
**Created:** [November 23, 2023, 1:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparameters-that-contribute-to-gpu-memory%2F63213 "2023-11-23T01:55:22Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Riyaz1003](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@Riyaz1003](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRiyaz1003)%5C
**Post date:** [November 23, 2023, 1:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparameters-that-contribute-to-gpu-memory%2F63213%2F1 "2023-11-23T01:55:22Z")

</div>

Hi,

I want to know exactly which all parameters are contributing to GPU Memory.

[https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.microsoft.com/en-us/research/uploads/prod/2020/09/dnnmem.pdf&ved=2ahUKEwjoy9zygNmCAxUGTGwGHUMtDUcQFnoECAsQAQ&usg=AOvVaw2Lu04KTTmjP39qHbyB\_Wa-](https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.microsoft.com/en-us/research/uploads/prod/2020/09/dnnmem.pdf&ved=2ahUKEwjoy9zygNmCAxUGTGwGHUMtDUcQFnoECAsQAQ&usg=AOvVaw2Lu04KTTmjP39qHbyB_Wa-)

Please explain with one simple example.

Let’s assume I have 1 layer Feed forward neural network. With input features size 100D and output layer with 2 units. Without bias

So total parameters that needs GPU :

1. For batch size of 8, weight Matrix of size 8_100_(100_2) ,weight gradient will 100_2

Is my understanding correct?