# Convert tokens and token-labels to string
**URL:**
**Author:**  [@Zack](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FZack)%5C
**Post date:** [November 16, 2020, 11:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconvert-tokens-and-token-labels-to-string%2F2086%2F1 "2020-11-16T11:24:13Z")
I am building a token classification model, and I am asking if thereâs a good way that I can transform the tokens and labels (each token has its label) to string, I know there is a tokenizer.convert\_tokens\_to\_string that convert tokens to strings, but I must also take into consideration the labels.
Any idea rather than creating my proper implementation !
---
**Author:**  [@rgwatwormhill](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frgwatwormhill)%5C
**Post date:** [November 17, 2020, 7:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconvert-tokens-and-token-labels-to-string%2F2086%2F2 "2020-11-17T19:57:55Z")
Hi Zack,
what form are the labels currently in?
I donât understand what you are building. Are the labels the classifications?
---
**Author:**  [@Zack](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FZack)%5C
**Post date:** [November 19, 2020, 11:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconvert-tokens-and-token-labels-to-string%2F2086%2F3 "2020-11-19T11:19:49Z")
Hi @rgwatwormhill, thank you for your reply,
letâs say that I have the bellow predictions on the sentences âthis is a test sentence using transformers model, the model used is CamemBertâ,
The problem is I want to go back to string tokens instead of tokens generated by the tokenizer and on the same time preserving the labels, for example using the word âtransformersâ is mapped to âtransfomerâ and âsâ, letâs say that transfomer is tagged as a label âmodelâ, I am facing a problem to preserve the labels, I tried implementing a simple function to decode tokens and labels at the same time but i found problems because I donât know very well how the tokenizer works,
I am asking if thereâs a way to decode and preserve tokens using the tokenizer class !
---
**Author:**  [@rgwatwormhill](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frgwatwormhill)%5C
**Post date:** [November 20, 2020, 12:12pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconvert-tokens-and-token-labels-to-string%2F2086%2F4 "2020-11-20T12:12:37Z")
Have you seen this tutorial [https://huggingface.co/transformers/custom\_datasets.html#token-classification-with-w-nut-emerging-entities](https://huggingface.co/transformers/custom_datasets.html#token-classification-with-w-nut-emerging-entities)
For more about the tokenizer, see [https://mccormickml.com/2019/07/22/BERT-fine-tuning/](https://mccormickml.com/2019/07/22/BERT-fine-tuning/)
Chris McCormick also has a tutorial specifically about NER, but youâd have to pay for that one and I havenât seen it.
---
**Author:**  [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [November 21, 2020, 10:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconvert-tokens-and-token-labels-to-string%2F2086%2F5 "2020-11-21T22:25:56Z")
Hi @Zack, the key trick contained in @rgwatwormhillâs first link is the following paragraph:
> Now we arrive at a common obstacle with using pre-trained models for token-level classification: many of the tokens in the W-NUT corpus are not in DistilBertâs vocabulary. Bert and many models like it use a method called WordPiece Tokenization, meaning that single words are split into multiple tokens such that each token is likely to be in the vocabulary. For example, DistilBertâs tokenizer would split the Twitter handle `@huggingface` into the tokens `['@', 'hugging', '##face']` . This is a problem for us because we have exactly one tag per token. If the tokenizer splits a token into multiple sub-tokens, then we will end up with a mismatch between our tokens and our labels.
>
> One way to handle this is to only train on the tag labels for the first subtoken of a split token. We can do this in đ¤ Transformers by setting the labels we wish to ignore to `-100` . In the example above, if the label for `@HuggingFace` is `3` (indexing `B-corporation` ), we would set the labels of `['@', 'hugging', '##face']` to `[3, -100, -100]` .
So basically what you should do is create a mapping for each named entity tag to some integer (e.g. PER â 0, LOC â 1, ORG â 2 etc) and then use -100 to label all the entityâs subtokens beyond the first one.
Hope that helps!
---
**Author:**  [@Zack](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FZack)%5C
**Post date:** [November 22, 2020, 10:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconvert-tokens-and-token-labels-to-string%2F2086%2F6 "2020-11-22T22:18:44Z")
Hi @lewtun,
Thank you for your reply, itâs so helpful.
I am not sure that I get the trick, why not just setting for the tags [3, 3, 3] for [â@â, âhuggingâ, â##faceâ] if @HuggingFace for example is the labels B-corporation with index 3 ?
Is it necessary to set the -100 as a value for labels to ignore or just use any other value, because for now I am using 0 as an index to tell the model to ignore this label ?
For the problem described in this discussion is something similar to this,
> [@Converting Word-level labels to WordPiece-level for Token Classification](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-word-level-labels-to-wordpiece-level-for-token-classification%2F2118)%3A
>
> Hi all, I am building a BertForTokenClassification model but I am having trouble figuring out how to format my dataset. I have already labeled my dataset with span labeling. So for example: sequence = âHugging Face Inc. is a company based in New York City. Its headquarters are in DUMBO, therefore very close to the Manhattan Bridge.â In my dataset, I would have this labeled by hand as: [(Hugging, B-org), (Face, I-org), (Inc., L-org), (is, O), (a, O), (company, O), ⌠(Bridge, L-org)] However,âŚ
but in the reverse from WordPiece level to word-level labels.
---
**Author:**  [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [November 25, 2020, 11:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconvert-tokens-and-token-labels-to-string%2F2086%2F7 "2020-11-25T11:05:19Z")
Hi @Zack, the reasons why we might not want to assign NER tags to every subword are nicely summarised by facehugger2020 in the other thread you linked to: [Converting Word-level labels to WordPiece-level for Token Classification](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconverting-word-level-labels-to-wordpiece-level-for-token-classification%2F2118%2F7)
Besides efficiency, one is also technically breaking the IOB2 format by having consecutive tokens be associated with say _B-ORG_. Personally, I find this confusing during debugging, so prefer the convention to only tag the first subword in an entity. Of course, this is just a convention and facehugger2020 links to a tutorial showing an alternative along the lines you suggested.
Now why the -100 value? I believe this comes from the PyTorch convention to ignore indices with -100 on the cross-entropy loss: [https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html)
You can override this value if you want, but then youâll need to tinker with the loss function of your Transformer model.
You should be careful about picking an index like 0 because the tokenizer probably has a predefined set of indices for _natural numbers_ and you might be overriding special tokens like `` this way.
---
**Author:**  [@lzhnicholas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flzhnicholas)%5C
**Post date:** [March 12, 2022, 7:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fconvert-tokens-and-token-labels-to-string%2F2086%2F8 "2022-03-12T07:58:12Z")
Hi @lewtun, So if when decode the id/label -100 for evaluation to compare the ground truth to our prediction sequence, do we just ignore the subsequent word piece?
`['@', 'hugging', '##face']` to `[3, -100, -100]` becomes `['@']` to `[3]`? Cos it seems that it may be another issue for alignment.