# Unable to run gguf model
**URL:**
**Author:**  [@hyadav22](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhyadav22)%5C
**Post date:** [January 6, 2025, 10:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-run-gguf-model%2F134446%2F1 "2025-01-06T10:45:12Z")
I have a server with 250gb of RAM(64 core), but no GPU. I want to run quantized llama3 70B model, i have tried [Llama-3.3-70B-Instruct-Q5\_K\_M.gguf](https://huggingface.co/unsloth/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q5_K_M.gguf) and [Llama-3.3-70B-Instruct-Q3\_K\_M.gguf](https://huggingface.co/unsloth/Llama-3.3-70B-Instruct-GGUF/blob/main/Llama-3.3-70B-Instruct-Q3_K_M.gguf). But they are failing during model load phase due to RAM limitation.
Code fails during model loading phase, it says “Converting and de-quantizing GGUF tensor…”, during this time I have checked RAM utilization from htop cmmnd, RAM is occupied completely, and the program will be killed .
I want to know, what does it mean by “Converting and de-quantizing GGUF tensor…” and is there any error in my code ?
Below is my code:
from transformers import pipeline
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline, AutoModelForSeq2SeqLM
# model initialization
model = ‘/vivo/genai\_poc/hf\_model/Llama-3.3-70B-Instruct-Q3\_K\_M’
path = ‘/vivo/genai\_poc/hf\_model/Llama-3.3-70B-Instruct-Q3\_K\_M/Llama-3.3-70B-Instruct-Q3\_K\_M.gguf’
tokenizer = AutoTokenizer.from\_pretrained(model, gguf\_file=path)
model = AutoModelForCausalLM.from\_pretrained(model, gguf\_file=path)
text2text = pipeline(“text2text-generation”, model=model, tokenizer=tokenizer)
# Get the answer
result = text2text(“translate English to French: New Delhi is India’s capital”)
print(f"Answer: {result[‘answer’]}")
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [January 6, 2025, 10:59am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Funable-to-run-gguf-model%2F134446%2F2 "2025-01-06T10:59:29Z")
I think that the current Transformers cannot execute GGUF in its quantized state, so Transformers will first de-quantize it. If you don’t specify the data type, it will be expanded to 32-bit floating point, so from Q4, the size will be 8 times larger…
If you are using GGUF from the CPU, I think that using Llamacpp or Ollama is easy, memory-saving, and fast.
If you need to train the model, you will need Transformers…
> **[Use Ollama with any GGUF Model on Hugging Face Hub](https://huggingface.co/docs/hub/ollama)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
>
>
> \### System Info
>
> Environment:
> OS: Ubuntu 24.04
> Python version: 3.11.8
> Trans…formers version: transformers==4.45.2
> Torch version: torch==2.3.0
> Model: Meta-Llama-3.1-70B-Q2\_K-GGUF - https://huggingface.co/Ffftdtd5dtft/Meta-Llama-3.1-70B-Q2\_K-GGUF
>
>
> \### Who can help?
>
> text models: @ArthurZucker
> generate: @zucchini-nlp
>
> \### Information
>
> \- \[X\] The official example scripts
>
>
> \### Reproduction
>
> Description:
> I am attempting to load a quantized GGUF model (Meta-Llama-3.1-70B-Q2\_K-GGUF) using the AutoTokenizer and AutoModel classes from Hugging Face transformers, but I am encountering a severe memory RAM usage during the de-quantization process (more than 90GB).
>
> \*\*Steps to Reproduce:\*\*
> \`\`\`
> tokenizer = AutoTokenizer.from\_pretrained("Ffftdtd5dtft/Meta-Llama-3.1-70B-Q2\_K-GGUF", trust\_remote\_code=True, gguf\_file="Meta-Llama-3.1-70B-Q2\_K-GGUF")
> model = AutoModel.from\_pretrained("Ffftdtd5dtft/Meta-Llama-3.1-70B-Q2\_K-GGUF", device\_map='auto', trust\_remote\_code=True, gguf\_file="Meta-Llama-3.1-70B-Q2\_K-GGUF")
> \`\`\`
>
> \*\*Observed Behavior:\*\*
> Memory Issues: When attempting to load the GGUF model (Meta-Llama-3.1-70B-Q2\_K), the system quickly exhausts available memory during the de-quantization. It use more than 90GB. I also try other different models such as https://huggingface.co/mradermacher/Meta-Llama-3.1-70B-GGUF
>
> \### Expected behavior
>
> \- Efficient Loading: I expect the GGUF model (like Meta-Llama-3.1-70B-Q2\_K-GGUF) to be loaded correctly using the transformers library, with a clear and efficient process for de-quantizing and loading the model without causing memory exhaustion, even in systems with less than 100GB of RAM.
>
> \- Support for Lower RAM Systems: Given that GGUF is a quantized format designed for efficiency, it would be ideal if transformers could either support a more memory-optimized loading process or allow partial model loading, enabling users with lower RAM systems (e.g., \< 100GB) to load and run these models effectively.
>
> \- Alternative Loading Method: If loading such models directly with AutoModel and AutoTokenizer is not possible due to GGUF-specific constraints, it would be helpful to have documentation or tools to convert these models into a compatible format (such as PyTorch) that can be handled within transformers.