# Llama 2 & 8K Training

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllama-2-8k-training%2F49415%3E%5C
**Category:** 🤗Transformers\
**Created:** [August 4, 2023, 2:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllama-2-8k-training%2F49415 "2023-08-04T02:26:35Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![delewis](https://avatars.discourse-cdn.com/v4/letter/d/bc8723/32.png) [@delewis](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdelewis)%5C
**Post date:** [August 4, 2023, 2:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fllama-2-8k-training%2F49415%2F1 "2023-08-04T02:26:35Z")

</div>

Hello,

I am using SFFTrainer & bitsandbytes to fine tune Llama-2-7B on a dataset where the input will consistently be 5K-8K tokens. I am using an A10G and have no problems with setting max\_seq\_length to 2K or 4K, but whenever I set it to 5K+ I run out of VRAM. Also, worth mentioning that I am using RoPE to hopefully accommodate the larger context.

Do I need to set max\_seq\_length to 8K to train effectively on this dataset? Also, what is the relationship of max\_seq\_length & num\_of\_sequences? Is there anyway to accomodate a larger sequence length like 8K in the the VRAM I have available? I thought I could do it by decreasing num\_of\_sequences, but that doesn’t seem to be have any effect on the amount of VRAM being reserved.

Thank you!