# Issues with offset\_mapping values
**URL:**
**Author:**  [@matus](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmatus)%5C
**Post date:** [March 8, 2021, 12:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissues-with-offset-mapping-values%2F4237%2F1 "2021-03-08T12:44:46Z")
Hi guys, I am trying to work with a FairSeq model converted to 🤗 but I have some issues with tokenizer. I am trying to fine-tune it for POS tagging so I have the text split to words already and I want to use the offset\_mapping to detect first token for each word. I do it like this:
```
tokenizer = RobertaTokenizerFast.from_pretrained('path', add_prefix_space=True)
ids = tokenizer([['drieme', 'drieme'], ['drieme']],
is_split_into_words=True,
padding=True,
return_offsets_mapping=True)
```
The tokenization looks like this:
```
['', 'Ä d', 'rieme', 'Ä d', 'rieme', '']
```
But the output from the command looks like this:
```
{
'input_ids': [[0, 543, 24209, 543, 24209, 2], [0, 543, 24209, 2, 1, 1]],
'attention_mask': [[1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 0, 0]],
'offset_mapping': [
[(0, 0), (0, 1), (1, 6), (1, 1), (1, 6), (0, 0)],
[(0, 0), (0, 1), (1, 6), (0, 0), (0, 0), (0, 0)]
]
}
```
Notice the offset mapping for the word _drieme_ in the first case. First word has mappings `(0, 1)` and `(1, 6)`. This looks reasonable, however the second _drieme_ is `(1, 1)` and `(1, 6)`. Suddenly, there is `1` at the first position. This `1` is there for all but first word for any sentence I try to parse. I feel like it might have something to do with handling the start of the sentence vs all the other words, but I am not sure how to solve this issue so that I have proper offset mappings. What am I doing wrong?
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [March 9, 2021, 11:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissues-with-offset-mapping-values%2F4237%2F2 "2021-03-09T23:11:35Z")
Thanks for reporting, it’s definitely a bug. Could you open an issue on [tokenizers](https://github.com/huggingface/tokenizers) with your snippet?
---
**Author:**  [@proxyht](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fproxyht)%5C
**Post date:** [April 10, 2021, 8:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissues-with-offset-mapping-values%2F4237%2F3 "2021-04-10T08:11:16Z")
Having similar issue as well. Is there a proper fix yet?
---
**Author:**  [@matus](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmatus)%5C
**Post date:** [April 10, 2021, 8:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissues-with-offset-mapping-values%2F4237%2F4 "2021-04-10T08:26:10Z")
Don’t know about the fix, but I solved it by detecting the special character that is used at the start of each word. In my case this was Ä . I used this instead of offset mapping to detect the start of the word.
---
**Author:**  [@mspinaci](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmspinaci)%5C
**Post date:** [February 15, 2022, 1:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissues-with-offset-mapping-values%2F4237%2F5 "2022-02-15T13:13:18Z")
Hello,
Is there any update on this topic? I am encountering the same issue with any tokenizer that has the `add_prefix_space=True` flag, which seems to be needed for example for RoBERTa tokenizers. A few more examples to exemplify the issue, using `words = ['hello', 'world', 'foo', 'bar', '32', '31290485']`:
- Using a non-RoBERTa tokenizer doesn’t show the issue and works as expected:
```auto
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
tokenizer(words, is_split_into_words=True, return_offsets_mapping=True)
# This returns:
# 'input_ids': [101, 7592, 2088, 29379, 3347, 3590, 21036, 21057, 18139, 2629, 102]
# 'offset_mapping': [(0, 0), (0, 5), (0, 5), (0, 3), (0, 3), (0, 2), (0, 3), (3, 5), (5, 7), (7, 8), (0, 0)]
# Running tokenizer.add_prefix_space would raise an AttributeError
```
in the above, as expected, special tokens have offsets (0, 0), beginning of words are (0, n), with n being the word/chunk length, continuations of words are of the form (n, m) with `n > 0` being the start of the chunk, and `m - n` the chunk length.
- However, for RoBERTa gives the following:
```auto
tokenizer = AutoTokenizer.from_pretrained('distilroberta-base', add_prefix_space=True)
tokenizer(words, is_split_into_words=True, return_offsets_mapping=True)
# This returns
# 'input_ids': [0, 20760, 232, 48769, 2003, 2107, 32490, 3248, 33467, 2]
# 'offset_mapping': [(0, 0), (0, 5), (1, 5), (1, 3), (1, 3), (1, 2), (1, 3), (3, 5), (5, 8), (0, 0)
tokenizer.add_prefix_space
# This returns True
```
The special tokens are still marked as (0, 0) offset, the first word is still of the form (0, n), the chunks from second on are still of the form (n, m), but each word beginning after the second is now of the form (1, n), with n still being the word/chunk length. So this looks to be the only exception to the rule that the length of the chunk is `m - n`, and that’s not good. I suppose the beginning `1` is because, as the flag name suggests, a space is added before the words and not taken into account when running the offsets\_mapping, but as suggested above that looks like a bug.
For my purpose (running NER) I can probably workaround the above by checking explicitly if the word is not the first word in the sentence, but the situation gets weirder when the chunks should be just 1 character long (so that, with the above logic, the offsets\_mapping should then be something like (1, 1), (1, 2)). For example, using `words = ['hello', 'world', 'foo', 'bar', '32', '31290485', 'üù']`, I am getting:
```auto
{'input_ids': [0, 20760, 232, 48769, 2003, 2107, 32490, 3248, 33467, 952, 4394, 3849, 9253, 2],
'offset_mapping': [(0, 0), (0, 5), (1, 5), (1, 3), (1, 3), (1, 2), (1, 3), (3, 5), (5, 8), (1, 1), (0, 1), (1, 2), (1, 2), (0, 0)]}
```
contradicting what I was expecting from above; I guess in this case, first the space that is implicitly added before `üù` gets the offset (1, 1), and then the letter `ü` (which doesn’t have any space before it anymore) correctly gets the offset (0, 1), but then somehow the offset for `ù` (which should indeed be (1, 2)`, being the second half of a two-character word) gets… doubled?
Edit: the weirdness in the last part seems to be explained by how the tokenizer works on Unicode characters, encoding them by piece. So `ü` is actually encoded as `ü` and `ù` is encoded as `ù`, as explained by running `tokenizer.convert_ids_to_tokens` on the output. It is made all a bit cleaner by running `tokenizer(' '.join(words))` and looking at its result, where indeed the tokens are duplicated, but also the offsets are correctly duplicated:
```auto
{'input_ids': [0, 20760, 232, 48769, 2003, 2107, 32490, 3248, 33467, 952, 4394, 3849, 9253, 2],
'offset_mapping': [(0, 0), (0, 5), (6, 11), (12, 15), (16, 19), (20, 22), (23, 26), (26, 28), (28, 31), (32, 33), (32, 33), (33, 34), (33, 34), (0, 0)]}
```
I guess that for now I’ll stick to using `tokenizer(' '.join(words))` instead of `tokenizer(words, is_split_into_words=True)`…