# Qwen tokenizer, omit Ġ from offset\_mapping
**URL:**
**Author:**  [@hfnlpmb](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhfnlpmb)%5C
**Post date:** [July 17, 2025, 5:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fqwen-tokenizer-omit-g-from-offset-mapping%2F163495%2F1 "2025-07-17T17:19:03Z")
I want to match words in text with the tokens created by Qwen’s tokenizer based on their text spans. I can get the text spans of the tokens from the `offset_mapping` but `Qwen3-Embedding-0.6B` `tokenizer` adds the `Ġ` symbol for `space`, which is also added in the `offset_mapping`. Can I omit the effect of `Ġ`/ `space`in the `offset_mapping` from the library, or should I do it manually?
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [July 18, 2025, 4:20am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fqwen-tokenizer-omit-g-from-offset-mapping%2F163495%2F2 "2025-07-18T04:20:19Z")
How about like this?
```py
from transformers import AutoTokenizer
model_ids = ["Qwen/Qwen3-Embedding-0.6B", "google-bert/bert-base-cased"]
for model_id in model_ids:
# 1. Prepare tokenizer and text
tokenizer = AutoTokenizer.from_pretrained(model_id, add_prefix_space=False)
text = "A girl is styling her hair."
# 2. Encode with offsets
encoding = tokenizer(text, return_offsets_mapping=True, add_special_tokens=False)
tokens = tokenizer.convert_ids_to_tokens(encoding["input_ids"])
offsets = encoding["offset_mapping"]
# 3. Adjust spans: skip the leading space for 'Ġ' tokens
corrected_offsets = []
for tok, (start, end) in zip(tokens, offsets):
if tok.startswith("Ġ") and start < len(text) and text[start] == " ":
start += 1
corrected_offsets.append((start, end))
print(model_id)
print(tokens)
print("Original:", offsets)
print("Corrected:", corrected_offsets)
"""
Qwen/Qwen3-Embedding-0.6B
['A', 'Ġgirl', 'Ġis', 'Ġstyling', 'Ġher', 'Ġhair', '.']
Original: [(0, 1), (1, 6), (6, 9), (9, 17), (17, 21), (21, 26), (26, 27)]
Corrected: [(0, 1), (2, 6), (7, 9), (10, 17), (18, 21), (22, 26), (26, 27)]
google-bert/bert-base-cased
['A', 'girl', 'is', 'styling', 'her', 'hair', '.']
Original: [(0, 1), (2, 6), (7, 9), (10, 17), (18, 21), (22, 26), (26, 27)]
Corrected: [(0, 1), (2, 6), (7, 9), (10, 17), (18, 21), (22, 26), (26, 27)]
"""
```
>
>
> \### System Info
>
> transformers 4.53.2, Ubuntu 22.04.4, python 3.11.13
>
> \### Who ca…n help?
>
> @ArthurZucker and @itazap There must be a problem with the \`offset\_mapping\` of Qwen3 \`tokenizer\`. The starting point in the text for each token, except the first and the last, is one position behind. I compared it with the BERT's \`tokenizer\`, which produces what is expected:
>
> \### Information
>
> \- \[\] The official example scripts
> \- \[x\] My own modified scripts
>
> \### Tasks
>
> \- \[\] An officially supported task in the \`examples\` folder (such as GLUE/SQuAD, ...)
> \- \[\] My own task or dataset (give details below)
>
> \### Reproduction
>
> \`\`\`
> sample\_text='A girl is styling her hair.'
> bert\_tokenizer = BertTokenizerFast.from\_pretrained('google-bert/bert-base-cased')
> bert\_encoding = bert\_tokenizer(
> text=sample\_text, add\_special\_tokens=False, return\_offsets\_mapping=True
> )
> print(bert\_encoding\['offset\_mapping'\])
> qwen\_tokenizer = AutoTokenizer.from\_pretrained('Qwen/Qwen3-Embedding-0.6B')
> qwen\_encoding = qwen\_tokenizer(
> text=sample\_text, add\_special\_tokens=False, return\_offsets\_mapping=True
> )
> print(qwen\_encoding\['offset\_mapping'\])
> \`\`\`
>
> \### Expected behavior
>
> \[(0, 1), (2, 6), (7, 9), (10, 17), (18, 21), (22, 26), (26, 27)\]
> \[(0, 1), (1, 6), (6, 9), (9, 17), (17, 21), (21, 26), (26, 27)\]
---
**Author:**  [@hfnlpmb](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhfnlpmb)%5C
**Post date:** [July 18, 2025, 9:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fqwen-tokenizer-omit-g-from-offset-mapping%2F163495%2F3 "2025-07-18T09:58:30Z")
@John6666 thank you for your reply.
Yes, if the text is clean, it can be done like this. If for example, there are multiple whites paces within the text, there will be tokens that represent white space only with the first space having been removed. This is why I asked if the `transformers` library has a functionality not to include the `Ġ` character in tokenization, or it should be done manually like the answer that you provided.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [July 18, 2025, 10:39am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fqwen-tokenizer-omit-g-from-offset-mapping%2F163495%2F4 "2025-07-18T10:39:53Z")
Looking at [the GitHub issue above](https://github.com/huggingface/transformers/issues/39401#issuecomment-3077842949) and [old posts](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissues-with-offset-mapping-values%2F4237)%2C it seems like there’s no way to do this…
It might be quicker to raise an issue in the Transoformers repo…