# Tokenizer unigram tutorial encode\_word function question

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-unigram-tutorial-encode-word-function-question%2F86224%3E%5C
**Category:** Beginners\
**Created:** [May 11, 2024, 9:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-unigram-tutorial-encode-word-function-question%2F86224 "2024-05-11T09:30:59Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nevermetyou](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nevermetyou/32/21618_2.png) [@Nevermetyou](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FNevermetyou)%5C
**Post date:** [May 11, 2024, 9:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-unigram-tutorial-encode-word-function-question%2F86224%2F1 "2024-05-11T09:30:59Z")

</div>

Hi

I just finish reading this tutorial on Unigram.

> **[Unigram tokenization - Hugging Face NLP Course](https://huggingface.co/learn/nlp-course/chapter6/7?fw=pt#implementing-unigram)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

And I have a question about this line in function `encode_word()`

```auto
best_segmentations = [{"start": 0, "score": 1}] + [
        {"start": None, "score": None} for _ in range(len(word))
    ]

```

As per my understanding, the score inside dictionary of the first list should be log(1) not 1 ???

Because in this line  
`score = model[token] + best_score_at_start`  
we are summing the log of probability.

So I suspect that `[{"start": 0, "score": 1}]` should be `[{"start": 0, "score": 0}]`

Can someone clarify me this matter?

Thanks