# Fine Tuning LLama 3.2 1B Quantized Memory Requirements
**URL:**
**Author:**  [@aman9608](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Faman9608)%5C
**Post date:** [November 11, 2024, 1:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-llama-3-2-1b-quantized-memory-requirements%2F117043%2F1 "2024-11-11T13:06:55Z")
Hi All!
I’m trying to fine tune a LLama 3.2 1B instruct model, that has been quantized during loading. But for some reason, the trainer errors out stating:
“OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 GiB. GPU 0 has a total capacity of 6.00 GiB of which 3.48 GiB is free.”
I’m not sure why this is the case that training a relatively small model is requiring 32GB of VRAM.
Would really appreciate any help if possible, code attached below:
```auto
quantization_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype=torch.float16,
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B-Instruct", num_labels = 23, quantization_config=quantization_config)
lora_config = LoraConfig(
r=4,
lora_alpha=8,
lora_dropout=0.1,
bias="none",
task_type=TaskType.CAUSAL_LM,
target_modules = ["q_proj", "k_proj", "v_proj"]
)
model = get_peft_model(model, lora_config)
training_args = TrainingArguments(
output_dir='./results',
num_train_epochs=1,
per_device_train_batch_size=1,
per_device_eval_batch_size=16,
warmup_steps=500,
weight_decay=0.01,
logging_dir='./logs',
logging_steps=10,
evaluation_strategy="epoch",
gradient_accumulation_steps=4,
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
```
---
**Author:**  [@BenjaminB](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBenjaminB)%5C
**Post date:** [November 29, 2024, 3:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-llama-3-2-1b-quantized-memory-requirements%2F117043%2F2 "2024-11-29T15:14:08Z")
With the given information, it’s hard to tell what the reason is for why so much memory is needed, there is nothing obviously wrong there. It would be helpful if you could share the full code and the full error message.
One common source of excessive memory usage can be in the data itself. If you have very long sequences, because of the quadratic memory requirement, OOMs can easily occur. You could check if setting a low `max_seq_length` helps to curb the memory usage.
---
**Author:**  [@DannyDias](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDannyDias)%5C
**Post date:** [January 23, 2025, 4:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-llama-3-2-1b-quantized-memory-requirements%2F117043%2F3 "2025-01-23T16:49:06Z")
Running LLaMA 3.2 locally requires adequate computational resources. Below are the recommended specifications:
Hardware:
GPU: NVIDIA GPU with CUDA support (16GB VRAM or higher recommended).
RAM: At least 32GB (64GB for larger models).
Storage: Minimum 50GB of free disk space for the model and dependencies.
Software:
Operating System: Linux (preferred), macOS, or Windows.
Python: Version 3.8 or higher.
CUDA Toolkit: Required for GPU acceleration (11.6 or newer).
---
**Author:**  [@rkapuaala](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frkapuaala)%5C
**Post date:** [June 16, 2025, 7:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-llama-3-2-1b-quantized-memory-requirements%2F117043%2F4 "2025-06-16T07:54:14Z")
Has anyone fixed this yet? I’m using a AMD Ryzen 5 4600G with Radeon Graphics on my ubuntu server.
I’ve got 64GB of ram, 2 asus Geforce rtx ti 16 GB 4060 gpus and 4 1T nvme ssds. I still keep getting those errors. I’ve tried configuring accelerate at the begining of the script, setting every environment variable I can find, and the CUCA still demands 32 GB.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [June 16, 2025, 2:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-llama-3-2-1b-quantized-memory-requirements%2F117043%2F5 "2025-06-16T14:52:43Z")
As BenjaminB also mentioned, it seems unusual that this model size and training parameters require this amount of RAM.
Therefore, there may be issues with the library or model, and the cause may lie in parts other than the visible parameters.
However, if this were a common issue, there would likely be many reports of it. It may be specific to certain conditions, such as a multi-GPU environment, a specific version of the library, or specific hardware.
> **[Fine tuning GPU memory requirements · ggml-org llama.cpp · Discussion #2904](https://github.com/ggml-org/llama.cpp/discussions/2904)**
>
> This question is more focused on on full fine tune memory requirements rather than low memory / efficient inference, but I'm hoping it'll be relevant / helpful to community members here especially ...
---
**Author:**  [@rkapuaala](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frkapuaala)%5C
**Post date:** [June 16, 2025, 3:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-llama-3-2-1b-quantized-memory-requirements%2F117043%2F6 "2025-06-16T15:58:21Z")
I’m wondering it maybe has something to do with using a local dataset instead of the hub, but that’s just a desperate conclusion after many hours of struggling with this issue. I’ll check out the link you provided hopefully that will provide some insight. Thanks.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [June 16, 2025, 4:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-llama-3-2-1b-quantized-memory-requirements%2F117043%2F7 "2025-06-16T16:08:39Z")
I see, the dataset could also be a possible cause…
Well, the best practices for datasets are probably available in this forum or on GitHub if you search for them…😅
Also, depending on the model, gradient checking may not be available (I think it should be available in Llama 3.2 1B, though…), and there may still be some potential bugs in multi-GPU environments.
When trying to isolate the issue, it’s usually faster to temporarily switch to a smaller, simpler model or dataset.
>
>
> Hi,
>
> I'm trying to supervised fine-tune a phi-1.5B model on a custom dataset w…ith the SFTTrainer, my script closely follows the \[sft\_llama2.py\](https://github.com/huggingface/trl/blob/main/examples/research\_projects/stack\_llama\_2/scripts/sft\_llama2.py).
>
> I'm training the model on 4x2080Ti (11G), the model 1.3B params should comfortable fit it the combined VRAM of the GPU's but I see a CUDA OOM errors when I start my training.
>
> my hyper parameters are as follows:
> \`\`\`python
> per\_device\_train\_batch\_size: Optional\[int\] = field(default=1, metadata={"help": "The batch size per GPU."})
> per\_device\_eval\_batch\_size: Optional\[int\] = field(default=1, metadata={"help": "The batch size per GPU for evaluation."})
> gradient\_accumulation\_steps: Optional\[int\] = field(default=8, metadata={"help": "The number of gradient accumulation steps."})
> gradient\_checkpointing: Optional\[bool\] = field(default=False, metadata={"help": "Whether to use gradient checkpointing."})
> \`\`\`
> furthermore , this is how I'm instantiating my model:
> \`\`\`python
> self.base\_model = AutoModelForCausalLM.from\_pretrained(
> pretrained\_model\_name\_or\_path=self.script\_args.model\_name,
> quantization\_config=self.bnb\_config,
> device\_map="auto",#{"": Accelerator().local\_process\_index},
> trust\_remote\_code=True,
> # torch\_dtype=torch.float16,
> # use\_flash\_attention\_2=False
> )
> \`\`\`
>
> I cannot use PEFT or Gradient Checkpointing as Phi models are not supported.
> [@LLama3-8B - FSDP + QLORA results in OOM with 4 A40's](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllama3-8b-fsdp-qlora-results-in-oom-with-4-a40s%2F92546)%3A
>
> Hardware: CPU: Xeon® E5-2630 v2 but limited to 16GB as this is what the vast.ai instance has. GPU: 4x A40 → Total of 180GB OS Linux python 3.10 cuda 12.2 packages: torch==2.3.1 transformers==4.41.2 peft==0.11.1 datasets==2.20.0 accelerate==0.31.0 evaluate==0.4.1 bitsandbytes==0.43.1 huggingface\_hub==0.23.4 trl==0.9.4 Issue Introduction Hi! I’m trying to fine-tune LLama3-8B on a summarization dataset of about 1500 instances. The dataset contains long documents, often over 8K tokens. I…
> [@Cuda OOM on 4 A6000s (142 GB of VRAM) even after using Zero3, Qlora, Accelerate, Max\_token\_length](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-oom-on-4-a6000s-142-gb-of-vram-even-after-using-zero3-qlora-accelerate-max-token-length%2F153978)%3A
>
> Trying to sft Qwen2.5vl-3b-instruct but I get this same error over and over again, I’ve looked at all the past threads and tried their solutions but its just not working. I don’t think downgrading to a smaller model will do any good because the error comes during attention which is quadratic with respect to N not model size. Maybe its an issue with my collate\_fn but I can’t find anything, I’ve even chomped down max\_token\_length to 1024 and its the same error so I feel like there’s something els…