# Trouble batch mapping dataset to tokenizer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrouble-batch-mapping-dataset-to-tokenizer%2F42244%3E%5C
**Category:** 🤗Datasets\
**Created:** [June 6, 2023, 10:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrouble-batch-mapping-dataset-to-tokenizer%2F42244 "2023-06-06T10:11:13Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![admarcosai](https://avatars.discourse-cdn.com/v4/letter/a/7ab992/32.png) [@admarcosai](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fadmarcosai)%5C
**Post date:** [June 6, 2023, 10:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrouble-batch-mapping-dataset-to-tokenizer%2F42244%2F1 "2023-06-06T10:11:13Z")

</div>

I am working on the WMT14 de\_en dataset; I have been trying to tokenize it using batch mapping but I seem to be doing something wrong or do not understand how the mapping function with batching works.

The following is my code:

```auto
from datasets import load_dataset
from transformers import GPT2TokenizerFast

dataset_de_en = load_dataset("wmt14", "de-en")
tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")

def tokenize(trans_sample):
    src_tokenized = tokenizer(trans_sample['en'])
    trg_tokenized = tokenizer(trans_sample['de'])
    return {'en': src_tokenized,
            'de': trg_tokenized}

tokenized = dataset_de_en['train']['translation'].map(tokenize, batched=True, batch_size=512)
tokenized

```

ERROR: essentially trans\_sample is a list and I would need a for loop to iterate through it to tokenize which to my understanding would mean I am not using map bathing properly here. Could someone please point me in the right direction as in how to do this properly?

```auto
AttributeError: 'list' object has no attribute 'map'

```

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [June 12, 2023, 8:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrouble-batch-mapping-dataset-to-tokenizer%2F42244%2F2 "2023-06-12T20:11:04Z")

</div>

You can find the answer by inspecting the processing function from the official translation notebook: [transformers/run\_translation\_no\_trainer.py at 41a8fa4e14ae14405a69efc65bfc21c9daa71c1a · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/41a8fa4e14ae14405a69efc65bfc21c9daa71c1a/examples/pytorch/translation/run_translation_no_trainer.py#L457-L474)