# Best practice to train LLMs on long sequences?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-practice-to-train-llms-on-long-sequences%2F111638%3E%5C
**Category:** 🤗Transformers\
**Created:** [October 12, 2024, 7:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-practice-to-train-llms-on-long-sequences%2F111638 "2024-10-12T19:52:31Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![oran-sh](https://avatars.discourse-cdn.com/v4/letter/o/a4c791/32.png) [@oran-sh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Foran-sh)%5C
**Post date:** [October 12, 2024, 7:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-practice-to-train-llms-on-long-sequences%2F111638%2F1 "2024-10-12T19:52:31Z")

</div>

Hello,

I am trying to fine tune gemma-2-2b on long sequences (4000-8000 tokens).  
Working with bf16 and 4 bit QLoRA still doesn’t fit to single 24GB GPU.  
I want to scale this up to multi-GPU instance but this means I have to split the model to several GPUs.  
What’s the best practice to training LLMs on long sequences?

Thank you