# Can't load tokenizer with added special tokens

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-load-tokenizer-with-added-special-tokens%2F16229%3E%5C
**Category:** 🤗Transformers\
**Created:** [March 29, 2022, 5:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-load-tokenizer-with-added-special-tokens%2F16229 "2022-03-29T17:30:33Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![ScandinavianMrT](https://avatars.discourse-cdn.com/v4/letter/s/c37758/32.png) [@ScandinavianMrT](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FScandinavianMrT)%5C
**Post date:** [March 29, 2022, 5:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-load-tokenizer-with-added-special-tokens%2F16229%2F1 "2022-03-29T17:30:33Z")

</div>

Hi! I have an annoying issue that I can’t find answers to anywhere online…  
Hope someone can help.

I am adding a few special tokens to a gpt2 tokenizer using the following code:

model\_name = “gpt2”  
model = AutoModelForCausalLM.from\_pretrained(model\_name)  
tokenizer = AutoTokenizer.from\_pretrained(model\_name)

special\_tokens = {  
‘eos\_token’: “\<|endoftext|\>”,  
‘bos\_token’: “\<|startoftext|\>”,  
‘additional\_special\_tokens’: ["\<|speaker1|\>", “\<|speaker2|\>”]  
}

tokenizer.pad\_token = tokenizer.eos\_token  
tokenizer.add\_special\_tokens(special\_tokens)  
vocab = tokenizer.get\_vocab()  
model.resize\_token\_embeddings(len(vocab))

I later save the tokenizer using:

model\_save\_name = ‘SARC\_gpt2\_prefinetune\_2.0’  
tokenizer.save\_pretrained(F"/content/drive/MyDrive/Colab Notebooks/saved\_models/{model\_save\_name}")

But when I load the model in a different script using:

tokenizer\_1 = AutoTokenizer.from\_pretrained(’/content/drive/MyDrive/Colab Notebooks/saved\_models/SARC\_gpt2\_prefinetune\_2.0’)

I get this error:

AssertionError: Non-consecutive added token ‘\<|startoftext|\>’ found. Should have index 50260 but has index 50257 in saved vocabulary.

Does anyone know what I’m doing wrong here?  
Thanks in advance