# Custom Dataset with Custom Tokenizer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-with-custom-tokenizer%2F7100%3E%5C
**Category:** 🤗Datasets\
**Created:** [June 23, 2021, 12:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-with-custom-tokenizer%2F7100 "2021-06-23T12:18:31Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![isarth](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/isarth/32/13332_2.png) [@isarth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fisarth)%5C
**Post date:** [June 23, 2021, 12:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-with-custom-tokenizer%2F7100%2F1 "2021-06-23T12:18:31Z")

</div>

I trained a BPE tokenizer using the wiki-text and now I’m trying to use this tokenizer on a custom dataset from a csv file. What I want to achieve is additional of feature columns in my dataset. But the dataset.map is giving error.

 ![Screenshot 2021-06-23 at 5.47.51 PM](https://us1.discourse-cdn.com/hellohellohello/original/2X/9/99822a36fe1d039de1d8f2b675f981bca377d8fa.png)

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [June 23, 2021, 12:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-with-custom-tokenizer%2F7100%2F2 "2021-06-23T12:26:16Z")

</div>

You should just use the tokenizer ` __call__ `: `tokenizer(example["text"])`.

---

<div class="post-metadata">

**Author:** ![isarth](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/isarth/32/13332_2.png) [@isarth](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fisarth)%5C
**Post date:** [June 23, 2021, 12:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-with-custom-tokenizer%2F7100%2F3 "2021-06-23T12:38:23Z")

</div>

When I train the tokenizer using this [Quicktour — tokenizers documentation](https://huggingface.co/docs/tokenizers/python/latest/quicktour.html)  
The call function is not implemented

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [June 23, 2021, 12:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcustom-dataset-with-custom-tokenizer%2F7100%2F4 "2021-06-23T12:50:41Z")

</div>

Oh, you should wrap your tokenizer in a `PreTrainedTokenizerFact` from the Transformers library (you can just pass your tokenizer with the `tokenizer_object` keyword argument).