# Fine-tuning Llama-7B

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-llama-7b%2F38455%3E%5C
**Category:** Models\
**Created:** [May 2, 2023, 5:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-llama-7b%2F38455 "2023-05-02T17:32:34Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![sp4912](https://avatars.discourse-cdn.com/v4/letter/s/e47c2d/32.png) [@sp4912](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsp4912)%5C
**Post date:** [May 2, 2023, 5:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-llama-7b%2F38455%2F1 "2023-05-02T17:32:34Z")

</div>

I am trying to fine-tune Llama-7B on a dataset of batch size 1 (so data is not the issue memory wise). I am using deepspeed with huggingface trainer. The issue I am facing is that deepspeed doesn’t let me set the model to a device separately (it does so automatically, causing OOM errors), but as a consequence, executing non-training/forward passes through the model/obtain logits take forever as they run on the CPU.

So what I am trying to do is withhold a GPU so that the model can execute forward calls on that GPU, after allowing deepspeed to modify it.

I have tried using deepspeed --include localhost:(GPUs I want deepspeed to use), but this sets CUDA\_VISIBLE\_DEVICES to exclude the withheld GPU I want to use, and deepspeed automatically uses all GPUs in --include. Is there any way I can solve this?

---

<div class="post-metadata">

**Author:** ![abdoelsayed](https://avatars.discourse-cdn.com/v4/letter/a/ac8455/32.png) [@abdoelsayed](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabdoelsayed)%5C
**Post date:** [May 2, 2023, 6:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-llama-7b%2F38455%2F2 "2023-05-02T18:07:44Z")

</div>

I had the same problem, but I solved it by using this rep. [GitHub - mallorbc/Finetune\_LLMs: Repo for fine-tuning GPTJ and other GPT models](https://github.com/mallorbc/Finetune_LLMs)  
You can listen to the description of how to use it on YouTube. [How To Fine-tune The LLaMA Models(GPT3 Alternative) - YouTube](https://www.youtube.com/watch?v=d4Cnv_g3GiI)

---

<div class="post-metadata">

**Author:** ![abdoelsayed](https://avatars.discourse-cdn.com/v4/letter/a/ac8455/32.png) [@abdoelsayed](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fabdoelsayed)%5C
**Post date:** [May 2, 2023, 6:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-llama-7b%2F38455%2F3 "2023-05-02T18:09:48Z")

</div>

Also, try to reduce the block size from 1024 to 128 and use batch size 64