# CUDA convert GUFF to CUDA GUFF
**URL:**
**Author:**  [@spencer1129](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fspencer1129)%5C
**Post date:** [December 18, 2024, 3:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-convert-guff-to-cuda-guff%2F132116%2F1 "2024-12-18T15:42:14Z")
Now I use llama.cpp with cuda.
And I load Meta-Llama-3.1-8B-Instruct-Q6\_K.gguf from hugging but it was not for gpu.
So I have to rebuild it for gpu. How can I do it?
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [December 18, 2024, 3:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-convert-guff-to-cuda-guff%2F132116%2F2 "2024-12-18T15:47:01Z")
I think GGUF itself can be loaded onto the GPU without any problems. It is possible that Llamacpp has been built without GPU support. Llamacpp is a software that is difficult to build properly with GPU support…
It is safer to use the pre-built version.
> **[Releases · abetlen/llama-cpp-python](https://github.com/abetlen/llama-cpp-python/releases)**
>
> Python bindings for llama.cpp. Contribute to abetlen/llama-cpp-python development by creating an account on GitHub.
If there is something wrong with GGUF itself, it is quicker to download it again.
> **[bartowski/Meta-Llama-3.1-8B-Instruct-GGUF · Hugging Face](https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
---
**Author:**  [@spencer1129](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fspencer1129)%5C
**Post date:** [December 18, 2024, 3:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-convert-guff-to-cuda-guff%2F132116%2F3 "2024-12-18T15:49:18Z")
I build llama.cpp with cuda enabled and load bartowski/Meta-Llama-3.1-8B-Instruct-GGUF]([bartowski/Meta-Llama-3.1-8B-Instruct-GGUF · Hugging Face](https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF).
so I lanuch server and it works. But it doesn’t use GPU.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [December 18, 2024, 3:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-convert-guff-to-cuda-guff%2F132116%2F4 "2024-12-18T15:55:55Z")
In that case, it is possible that there is not enough VRAM, or that parameters such as n\_gpu\_layers, n\_ctx are not set appropriately. For information on settings, please refer to the following. If you try using a very small GGUF, you should be able to tell whether the problem is with VRAM or not.
Also, even if you build with CUDA specified, there are many cases where CUDA is not actually enabled.
>
>
> I installed llamacpp using the instructions below:
> CMAKE\_ARGS="-DLLAMA\_CUBLAS=o…n" pip install llama-cpp-python
> the speed:
> llama\_print\_timings: eval time = 81.91 ms / 2 runs ( 40.95 ms per token, 1.02 tokens per second)
>
> I installed llamacpp using the instructions below:
> pip install llama-cpp-python
> the speed:
> llama\_print\_timings: eval time = 81.91 ms / 2 runs ( 40.95 ms per token, 30.01 tokens per second)
>
> My code is as follows:
> result = self.model(
> prompt, # Prompt
> # max\_tokens=nt, # Generate up to 32 tokens
> # stop=\["Q:", "\\n"\], # Stop generating just before the model would generate a new question
> # echo=True # Echo the prompt back in the output
> )
>
> So how can i use GPU to speed up?
---
**Author:**  [@spencer1129](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fspencer1129)%5C
**Post date:** [December 18, 2024, 3:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-convert-guff-to-cuda-guff%2F132116%2F5 "2024-12-18T15:59:23Z")
yah I do it. And I use llama-server. I think it requires additional options for cuda. But I don’t know exactly!
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [December 18, 2024, 4:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-convert-guff-to-cuda-guff%2F132116%2F6 "2024-12-18T16:06:44Z")
I’ve never used it in server mode, but it seems that you can specify options using the method below. I think the effect is the same.
>
>
>
> I installed llamacpp using the instructions below:
> CMAKE\_ARGS="-DLLAMA\_CUBLAS=o…n" pip install llama-cpp-python
> the speed:
> llama\_print\_timings: eval time = 81.91 ms / 2 runs ( 40.95 ms per token, 1.02 tokens per second)
>
> I installed llamacpp using the instructions below:
> pip install llama-cpp-python
> the speed:
> llama\_print\_timings: eval time = 81.91 ms / 2 runs ( 40.95 ms per token, 30.01 tokens per second)
>
> My code is as follows:
> result = self.model(
> prompt, # Prompt
> # max\_tokens=nt, # Generate up to 32 tokens
> # stop=\["Q:", "\\n"\], # Stop generating just before the model would generate a new question
> # echo=True # Echo the prompt back in the output
> )
>
> So how can i use GPU to speed up?
---
**Author:**  [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [December 19, 2024, 4:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcuda-convert-guff-to-cuda-guff%2F132116%2F7 "2024-12-19T04:07:14Z")
Closed on December 19, 2024, 4:07am UTC
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.