# Use LongT5 model with T5 checkpoint

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fuse-longt5-model-with-t5-checkpoint%2F29739%3E%5C
**Category:** 🤗Transformers\
**Created:** [January 16, 2023, 9:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fuse-longt5-model-with-t5-checkpoint%2F29739 "2023-01-16T09:36:07Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![stecklin](https://avatars.discourse-cdn.com/v4/letter/s/b3f665/32.png) [@stecklin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fstecklin)%5C
**Post date:** [January 16, 2023, 9:36am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fuse-longt5-model-with-t5-checkpoint%2F29739%2F1 "2023-01-16T09:36:08Z")

</div>

The LongT5 paper claims LongT5 is compatible with T5 checkpoints.

> We experiment with two attention mechanism variations for LongT5 […]: (1) Local Attention and (2) Transient Global Attention(TGlobal).  
> Both variations preserve several properties of T5: […] compatibility with T5 checkpoints.

How do I accomplish this with transformers?

```python
tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = LongT5ForConditionalGeneration.from_pretrained("t5-small")

```

The console output tells me there is a difference in naming:

- encoder.block.0.layer.0.SelfAttention.o.weight for T5
- encoder.block.0.layer.0.LocalSelfAttention.o.weight for LongT5

Is there a way to map the weight names when loading a checkpoint? Or should I download the checkpoint and modify the files before loading it?

Background:

For my use case, I have long input sequences and frequently run into memory problems. LongT5 looks like a good candidate, but the smallest officially released checkpoint is `google/long-t5-<type>-base`. That’s why I would like to try `t5-small` checkpoint instead.