# Tokenizer is not defined

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-is-not-defined%2F39231%3E%5C
**Category:** 🤗Transformers\
**Created:** [May 9, 2023, 10:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-is-not-defined%2F39231 "2023-05-09T10:10:29Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![sbansal5](https://avatars.discourse-cdn.com/v4/letter/s/22d042/32.png) [@sbansal5](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsbansal5)%5C
**Post date:** [May 9, 2023, 10:10am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-is-not-defined%2F39231%2F1 "2023-05-09T10:10:30Z")

</div>

I tried everything but getting tokenizer error, here is my simple code:  
from datasets import Dataset, load\_dataset  
from transformers import AutoTokenizer, BertTokenizer,DataCollatorForLanguageModeling, Trainer, TrainingArguments, AutoModelForCausalLM

context\_dataset = Dataset.from\_dict({  
“text”: [“context sentence 1”, “context sentence 2”, “context sentence 3”]  
})  
print(context\_dataset)

# Load a pre-trained tokenizer

tokenizer = AutoTokenizer.from\_pretrained(“distilbert-base-uncased-distilled-squad”)  
print(tokenizer)

# Tokenize your dataset

def tokenize\_function(examples):  
return tokenizer(examples[“text”], padding=“max\_length”, truncation=True)

tokenized\_datasets = context\_dataset.map(tokenize\_function, batched=True, num\_proc=3, remove\_columns=[“text”])

---

<div class="post-metadata">

**Author:** ![hangjoo](https://avatars.discourse-cdn.com/v4/letter/h/d07c76/32.png) [@hangjoo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhangjoo)%5C
**Post date:** [May 9, 2023, 2:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-is-not-defined%2F39231%2F2 "2023-05-09T14:46:34Z")

</div>

The reason of that error is that the tokenizer was not declared within the `tokenize_function` function.  
A simple solution is to declare the tokenizer instance inside the `tokenize_function` function, but this is inefficient because it creates a new tokenizer instance every time the function is called. Other simple solution is to add tokenizer argument on `tokenize_function` function, but the `function` argument of Datasets’s `map` method only accepts one argument(examples).

There are two solutions to solve this problem, as described below.

### solution 1. Use TokenizerWrapper class

```python
class TokenizerWrapper:
    def __init__ (self, tokenizer):
        self.tokenizer = tokenizer
    
    def tokenize_function(self, examples):
        return self.tokenizer(
            examples["text"],
            padding="max_length",
            truncation=True,
        )

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased-distilled-squad")
tokenizer_wrapper = TokenizerWrapper(tokenizer)
tokenized_dataset = context_dataset.map(tokenizer_wrapper.tokenize_function, batched=True, num_proc=3, remove_columns=["text"])

```

### solution 2. Use [partial](https://docs.python.org/3/library/functools.html?highlight=partial#functools.partial) function

```python
from functools import partial

from transformers import AutoTokenizer

def tokenize_function(tokenizer, examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased-distilled-squad")
partial_tokenize_function = partial(tokenize_function, tokenizer)
tokenized_dataset = context_dataset.map(partial_tokenize_function, batched=True, num_proc=3, remove_columns=["text"])

```

---

<div class="post-metadata">

**Author:** ![satyroffrost](https://avatars.discourse-cdn.com/v4/letter/s/4bbf92/32.png) [@satyroffrost](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsatyroffrost)%5C
**Post date:** [November 13, 2023, 6:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-is-not-defined%2F39231%2F3 "2023-11-13T18:44:08Z")

</div>

Both this methods does not work.

```auto
from datasets import load_dataset

eli5 = load_dataset("eli5", split="train_asks[:500]")
eli5 = eli5.train_test_split(test_size=0.2)
eli5 = eli5.flatten()

class TokenizerWrapper:
    def __init__ (self, tokenizer):
        self.tokenizer = tokenizer
    
    def tokenize_function(self, examples):
        '''cut off more then 1024 tokens, this is model restriction'''
        return tokenizer([" ".join(x[:1024]) for x in examples["answers.text"]])

from transformers import AutoTokenizer
checkpoint = "distilgpt2"

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
print(tokenizer)
tokenizer_wrapper = TokenizerWrapper(tokenizer)
tokenized_eli5 = eli5.map(tokenizer_wrapper.tokenize_function,
    batched=True,
    num_proc=4,
    remove_columns=eli5["train"].column_names,)

```

## return Using sep\_token, but it is not set yet. Using pad\_token, but it is not set yet. Using cls\_token, but it is not set yet. Using mask\_token, but it is not set yet. GPT2TokenizerFast(name\_or\_path=‘distilgpt2’, vocab\_size=50257, model\_max\_length=1024, is\_fast=True, padding\_side=‘right’, truncation\_side=‘right’, special\_tokens={‘bos\_token’: ‘\<|endoftext|\>’, ‘eos\_token’: ‘\<|endoftext|\>’, ‘unk\_token’: ‘\<|endoftext|\>’}, clean\_up\_tokenization\_spaces=True), added\_tokens\_decoder={ 50256: AddedToken(“\<|endoftext|\>”, rstrip=False, lstrip=False, single\_word=False, normalized=False, special=True), } Map (num\_proc=4): 0% 0/400 [00:02\<?, ? examples/s]

RemoteTraceback Traceback (most recent call last)  
RemoteTraceback:  
“”"  
Traceback (most recent call last):  
File “C:\Users\satyr\anaconda3\lib\site-packages\multiprocess\pool.py”, line 125, in worker  
result = (True, func(\*args, \*\*kwds))  
File “C:\Users\satyr\anaconda3\lib\site-packages\datasets\utils\py\_utils.py”, line 1354, in \_write\_generator\_to\_queue  
for i, result in enumerate(func(\*\*kwargs)):  
File “C:\Users\satyr\anaconda3\lib\site-packages\datasets\arrow\_dataset.py”, line 3474, in \_map\_single  
batch = apply\_function\_on\_filtered\_inputs(  
File “C:\Users\satyr\anaconda3\lib\site-packages\datasets\arrow\_dataset.py”, line 3353, in apply\_function\_on\_filtered\_inputs  
processed\_inputs = function(\*fn\_args, \*additional\_args, \*\*fn\_kwargs)  
File “C:\Users\satyr\AppData\Local\Temp\ipykernel\_7500\3052597454.py”, line 13, in tokenize\_function  
NameError: name ‘tokenizer’ is not defined  
“”"

```auto
from datasets import load_dataset

eli5 = load_dataset("eli5", split="train_asks[:500]")
eli5 = eli5.train_test_split(test_size=0.2)
eli5 = eli5.flatten()

from functools import partial
from transformers import AutoTokenizer
def tokenize_function(tokenizer, examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)
checkpoint = "distilgpt2"

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
print(tokenizer)
partial_tokenize_function = partial(tokenize_function, tokenizer)
tokenized_eli5 = eli5.map(tokenizer_wrapper.tokenize_function,
    batched=True,
    num_proc=4,
    remove_columns=eli5["train"].column_names,)

```

return

## Using sep\_token, but it is not set yet. Using pad\_token, but it is not set yet. Using cls\_token, but it is not set yet. Using mask\_token, but it is not set yet. GPT2TokenizerFast(name\_or\_path=‘distilgpt2’, vocab\_size=50257, model\_max\_length=1024, is\_fast=True, padding\_side=‘right’, truncation\_side=‘right’, special\_tokens={‘bos\_token’: ‘\<|endoftext|\>’, ‘eos\_token’: ‘\<|endoftext|\>’, ‘unk\_token’: ‘\<|endoftext|\>’}, clean\_up\_tokenization\_spaces=True), added\_tokens\_decoder={ 50256: AddedToken(“\<|endoftext|\>”, rstrip=False, lstrip=False, single\_word=False, normalized=False, special=True), } Map (num\_proc=4): 0% 0/400 [00:02\<?, ? examples/s]

RemoteTraceback Traceback (most recent call last)  
RemoteTraceback:  
“”"  
Traceback (most recent call last):  
File “C:\Users\satyr\anaconda3\lib\site-packages\multiprocess\pool.py”, line 125, in worker  
result = (True, func(\*args, \*\*kwds))  
File “C:\Users\satyr\anaconda3\lib\site-packages\datasets\utils\py\_utils.py”, line 1354, in \_write\_generator\_to\_queue  
for i, result in enumerate(func(\*\*kwargs)):  
File “C:\Users\satyr\anaconda3\lib\site-packages\datasets\arrow\_dataset.py”, line 3474, in \_map\_single  
batch = apply\_function\_on\_filtered\_inputs(  
File “C:\Users\satyr\anaconda3\lib\site-packages\datasets\arrow\_dataset.py”, line 3353, in apply\_function\_on\_filtered\_inputs  
processed\_inputs = function(\*fn\_args, \*additional\_args, \*\*fn\_kwargs)  
File “C:\Users\satyr\AppData\Local\Temp\ipykernel\_7500\3052597454.py”, line 13, in tokenize\_function  
NameError: name ‘tokenizer’ is not defined  
“”"

---

<div class="post-metadata">

**Author:** ![satyroffrost](https://avatars.discourse-cdn.com/v4/letter/s/4bbf92/32.png) [@satyroffrost](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsatyroffrost)%5C
**Post date:** [November 13, 2023, 6:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-is-not-defined%2F39231%2F4 "2023-11-13T18:53:54Z")

</div>

Only method i have find is to incapsulate all variables in mapping function:

```auto
from datasets import load_dataset

eli5 = load_dataset("eli5", split="train_asks[:500]")
eli5 = eli5.train_test_split(test_size=0.2)
eli5 = eli5.flatten()

def tokenize_function(examples):
    from transformers import AutoTokenizer
    checkpoint = "distilgpt2"
    tokenizer = AutoTokenizer.from_pretrained(checkpoint)
    return tokenizer([" ".join(x) for x in examples["answers.text"]])

tokenized_eli5 = eli5.map(tokenize_function,
    batched=True,
    num_proc=4,
    remove_columns=eli5["train"].column_names,)

```

You have to make hard decisions to work with HF.

---

<div class="post-metadata">

**Author:** ![tunggad](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@tunggad](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftunggad)%5C
**Post date:** [December 22, 2023, 9:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-is-not-defined%2F39231%2F5 "2023-12-22T21:54:13Z")

</div>

I got into the reported error while trying to follow the guide [Causal language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling) at step preprocess\_function and applying the solution 1 - TokenizerWrapper helped! Thank you hangjoo!

---

<div class="post-metadata">

**Author:** ![remcbdx](https://avatars.discourse-cdn.com/v4/letter/r/ac91a4/32.png) [@remcbdx](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fremcbdx)%5C
**Post date:** [March 19, 2024, 7:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftokenizer-is-not-defined%2F39231%2F6 "2024-03-19T19:02:04Z")

</div>

Still not working? I am trying to finetune whisper and I cant because of the tokenizer call. This is really painful