# NER model fine tuning with labeled spans
**URL:**
**Author:**  [@astarostap](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fastarostap)%5C
**Post date:** [October 9, 2021, 2:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fner-model-fine-tuning-with-labeled-spans%2F10633%2F1 "2021-10-09T02:01:56Z")
Hi!
I’m looking to fine-tune an NER model (dslim/bert-base-NER-uncased) with my own data.
My annotations are of this form: for each example I have a piece of raw text (str) and a list of annotated spans of this form: {start\_index: int, end\_index: int, tag: str}
However, to fine-tune the NER model, I need to prepare X (tokens) and Y (token tags) for each example. So, those spans have to be translated into token tags, matching the model’s tokenizer.
Hope that makes sense.
Is there a way to handle this? Or, what would you recommend?
Thanks!
---
**Author:**  [@astarostap](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fastarostap)%5C
**Post date:** [October 23, 2021, 4:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fner-model-fine-tuning-with-labeled-spans%2F10633%2F2 "2021-10-23T16:02:26Z")
Hi folks! Does this make sense?
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [October 25, 2021, 9:43am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fner-model-fine-tuning-with-labeled-spans%2F10633%2F3 "2021-10-25T09:43:31Z")
Hi,
Yes that’s the right approach. You can take a look at Spacy’s [`offsets_to_biluo_tags`](https://spacy.io/api/top-level#offsets_to_biluo_tags) method. It’s great to convert character index-level annotations to token annotations (in BILOU-format, which is a bit more exotic than IOB).
---
**Author:**  [@astarostap](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fastarostap)%5C
**Post date:** [October 25, 2021, 5:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fner-model-fine-tuning-with-labeled-spans%2F10633%2F4 "2021-10-25T17:09:25Z")
Thank you @nielsr!
The problem with that is that offsets\_to\_biluo\_tags uses some spacy tokenizer right? And they use a different tokenizer from Huggingface BERT tokenizer for example. So don’t I need something that uses HF’s BERT tokenizer to go from spans to token labels?
---
**Author:**  [@zirui3](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzirui3)%5C
**Post date:** [October 26, 2021, 1:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fner-model-fine-tuning-with-labeled-spans%2F10633%2F5 "2021-10-26T13:18:49Z")
you can convert the `span` to `BIOS` tags(other tag schema is similar) like this:
```auto
def convert_ents_to_bio(text, ents):
tags = ["O"] * len(text)
for ent in ents:
start, end, label = ent["start"], ent["end"], ent["label"]
if start == end:
tags[start] = 'S-'+ label
else:
tags[start] = 'B-' + label
tags[start+1: end + 1] = ['I-'+label]*(end - start)
return tags
input:
text = ["tom", "lives", "in", "Los", "Angeles"]
ents = [
{"start":0, "end": 0, "label": "PERSON"},
{"start":3, "end": 4, "label": "ADDRESS"},
]
convert_ents_to_bio(test, ents)
output:
['S-PERSON', 'O', 'O', 'B-ADDRESS', 'I-ADDRESS']
```
---
**Author:**  [@ben-epstein](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fben-epstein)%5C
**Post date:** [May 7, 2023, 3:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fner-model-fine-tuning-with-labeled-spans%2F10633%2F6 "2023-05-07T15:08:50Z")
Hi @astarostap I understand your issue! I was struggling with the same thing, as I needed the tokens and tags to align to my chosen tokenizer, which was subject to change.
I created a python package to handle this for me because I couldn’t find it anywhere online: [GitHub - Ben-Epstein/spacy-to-hf: A simple converter from SpaCy Entities (Spans) to Huggingface BILOU formatted data (tokens and ner\_tags)](https://github.com/Ben-Epstein/spacy-to-hf)
I hope it’s helpful!