# How to train with very long sequences?
**URL:**
**Author:**  [@zeev55](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzeev55)%5C
**Post date:** [May 18, 2022, 6:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-train-with-very-long-sequences%2F18088%2F1 "2022-05-18T06:52:02Z")
Hi, I’m trying to fine-tune a classifier on very long sequences that can reach up to 6000000 characters and even more. I started with GPT-2 model, and had to truncate the sequences to the limit of the model, which is 1024 if I understood correctly.
Is there a way for me to train this or a different model on longer sequences?
---
**Author:**  [@mab](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmab)%5C
**Post date:** [May 18, 2022, 8:12pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-train-with-very-long-sequences%2F18088%2F2 "2022-05-18T20:12:43Z")
There is longformer, a transformer created for long sequences. I think in their paper they claim to be able to attend tens of thousands of tokens.
---
**Author:**  [@zeev55](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzeev55)%5C
**Post date:** [May 20, 2022, 6:52am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-train-with-very-long-sequences%2F18088%2F3 "2022-05-20T06:52:02Z")
thank you!