# Best Small LLM For Rag
**URL:**
**Author:**  [@Siyam](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSiyam)%5C
**Post date:** [March 4, 2025, 9:22am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-small-llm-for-rag%2F143971%2F1 "2025-03-04T09:22:20Z")
Which Is the best small model (3b) for rag, I am building a rag and using mistral-nemo 12b for it, i have tested many other model but not getting expected output like mistral nemo providing, nemo exactly follow system prompt but i can’t find any 3b model which exactly follow my system prompt, its normal that nemo is 12b model so it works better than any 3b model, but in my case i don’t want my model to have a large knowledge base outside my domain (200 pdf’s), and i want it to be super fast …
i am currently using Ollama ,please suggest the best 2-4b model for rag , smaller is better
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [March 4, 2025, 10:14am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-small-llm-for-rag%2F143971%2F2 "2025-03-04T10:14:39Z")
If you want a model that is as versatile as possible in that size range, I recommend these models.
> **[Qwen/Qwen2.5-3B-Instruct · Hugging Face](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
> **[meta-llama/Llama-3.2-3B-Instruct · Hugging Face](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
> **[google/gemma-2-2b-it · Hugging Face](https://huggingface.co/google/gemma-2-2b-it)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
---
**Author:**  [@anon69948774](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fanon69948774)%5C
**Post date:** [March 13, 2025, 12:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-small-llm-for-rag%2F143971%2F3 "2025-03-13T12:02:51Z")
Among the 7 or 8B models, [Ministral instruct 2410 GGUF](https://huggingface.co/bartowski/Ministral-8B-Instruct-2410-GGUF) is the best for me in french (IQ4 XS is small), so it’s probably also the best among the 3Bs.
For local PDF GPT4all is interesting, LocalDocs is efficient.
---
**Author:**  [@Akjava](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAkjava)%5C
**Post date:** [March 13, 2025, 12:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-small-llm-for-rag%2F143971%2F4 "2025-03-13T12:27:42Z")
I’m using granite3.2:8b for rag.granite3.2:2b is good as 2B.
but I’m not sure if the model can understand the system prompt provided by you.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [March 13, 2025, 2:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-small-llm-for-rag%2F143971%2F5 "2025-03-13T14:10:08Z")
Gemma 3 has been released, and 4B and 1B are in the lineup.
This seems to fit this use case.
> **[Google's Gemma models family - a google Collection](https://huggingface.co/collections/google/googles-gemma-models-family-675bfd70e574a62dd0e406bd)**
>
> Unlock the magic of AI with handpicked models, awesome datasets, papers, and mind-blowing Spaces from google