# BART Tokenizer tokenises same word differently?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbart-tokenizer-tokenises-same-word-differently%2F21835%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [August 19, 2022, 4:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbart-tokenizer-tokenises-same-word-differently%2F21835 "2022-08-19T16:53:35Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![AndreaSottana](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@AndreaSottana](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAndreaSottana)%5C
**Post date:** [August 19, 2022, 4:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbart-tokenizer-tokenises-same-word-differently%2F21835%2F1 "2022-08-19T16:53:35Z")

</div>

Hello,  
I have noticed that if I tokenize a full text with many sentences, I sometimes get a different number of tokens than if I tokenise each sentence individually and add up the tokens. I have done some debugging and have this small reproducible example to show the issue

```auto
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('facebook/bart-large-cnn')

print(tokenizer.tokenize("Thames is a river"))
print(tokenizer.tokenize("We are in London. Thames is a river"))

```

I get the following output

```auto
['Th', 'ames', 'Ġis', 'Ġa', 'Ġriver']
['We', 'Ġare', 'Ġin', 'ĠLondon', '.', 'ĠThames', 'Ġis', 'Ġa', 'Ġriver']

```

I would like to understand why the word `Thames` has been split into two tokens when it’s at the start of sequence, whereas it’s a single word if it’s not at the start of sequence. I have noticed this behaviour is very frequent and, assuming it’s not a bug, I would like to understand why the BART tokeniser behaves like this.

Many thanks

---

<div class="post-metadata">

**Author:** ![AndreaSottana](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@AndreaSottana](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAndreaSottana)%5C
**Post date:** [August 24, 2022, 2:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbart-tokenizer-tokenises-same-word-differently%2F21835%2F2 "2022-08-24T14:43:37Z")

</div>

UPDATE:  
For those interested, the reply to this question has now been provided by a StackOverflow user [here](https://stackoverflow.com/questions/73459649/bart-tokenizer-tokenises-same-word-differently/73459967#73459967)