# Preprocessing of dataset

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpreprocessing-of-dataset%2F81086%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [April 10, 2024, 1:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpreprocessing-of-dataset%2F81086 "2024-04-10T13:54:11Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![boringblobking](https://avatars.discourse-cdn.com/v4/letter/b/da6949/32.png) [@boringblobking](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fboringblobking)%5C
**Post date:** [April 10, 2024, 1:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpreprocessing-of-dataset%2F81086%2F1 "2024-04-10T13:54:12Z")

</div>

I’m going through [this](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/question_answering.ipynb) notebook and below is a gist of some of it:

```auto
datasets = load_dataset("squad_v2")
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
example = datasets["train"][i]

tokenized_example = tokenizer(
    example["question"],
    example["context"],
    max_length=max_length,
    truncation="only_second",
    return_overflowing_tokens=True,
    return_offsets_mapping=True,
    stride=doc_stride
)

sequence_ids = tokenized_example.sequence_ids()

answers = example["answers"]
start_char = answers["answer_start"][0]
end_char = start_char + len(answers["text"][0])
# Start token index of the current span in the text.
token_start_index = 0
while sequence_ids[token_start_index] != 1:
    token_start_index += 1

# End token index of the current span in the text.
token_end_index = len(tokenized_example["input_ids"][0]) - 1
while sequence_ids[token_end_index] != 1:
    token_end_index -= 1

```

So as you can see, return\_overflowing\_tokens is True so the tokenized\_examples input\_ids are a list of lists so tokenized\_example[“input\_ids”][0] would contain just the first segment of the context for that sample.

So then to get token\_end\_index, the code starts at len(tokenized\_example[“input\_ids”][0]) - 1 and then moves backwards until it finds a 1. But surely len(tokenized\_example[“input\_ids”][0]) - 1 isn’t going to be the end of the context because len(tokenized\_example[“input\_ids”][0]) is only going to give you the length of the first segment of the context?