# \[Transformer\] how to tokenize nested object dataset? **URL:** **Author:** ![Honam](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/honam/32/12683_2.png) [@Honam](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHonam)%5C **Post date:** [November 23, 2022, 3:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftransformer-how-to-tokenize-nested-object-dataset%2F26560%2F1 "2022-11-23T03:56:14Z") e.g., I am trying to tokenize [ParlAI/blended\_skill\_talk · Datasets at Hugging Face](https://huggingface.co/datasets/blended_skill_talk/viewer/default/train) with BlenderbotTokenizer example data record: ![Screen Shot 2022-11-22 at 19.53.50](https://us1.discourse-cdn.com/hellohellohello/original/2X/8/8d88b0787fc94dede881fb69d37355587cd08543.png) and “suggestions” column structure: ```auto { "convai2": ["i am a professional translator .", "i know about 50 of the language"], "wizard_of_wikipedia": ["That means you must be in the Italian national football team.", "Thanks to the internet people are often exposed to multiple languages"] } ``` my tokenizing code but got some error because ““suggestions”” column has a nested structure like above example. ```auto dataset_train = dataset_train.map(lambda examples: tokenizer(examples["personas"], examples["context"], examples["previous_utterance"], examples["free_messages"], examples["guided_messages"], **examples["suggestions"],** truncation=True), batched=True) ``` ## error ```auto -------------------------------------------------------------------------- ValueError Traceback (most recent call last) ValueError: [{'convai2': ["i love acting ! i'll be famous someday . what do you do ?", 'no no kids , might get some though . one day', 'that is great . i am going to a concert later', '15 and 17 , two boys sooo fun', 'they really are . and a handful at times', 'it can be sometimes . i bet being a doctor is a lot of work too .'], 'empathetic_dialogues': ['Any favorite actors?', 'One day.', 'How long must you attend school?', '4 and 5 and I have a teenager', 'They are most of the time!', "Oh. I don't know how medical school works. I am studying srt history."], 'wizard_of_wikipedia': ['I would like to develop my acting skills. What are some tips you have to not get nervous?', 'I will still wimp out. i want to be famous like the rolling stones though.', 'good', "Close to 30! I just always have to put in a ton of work when mother's day comes around haha", 'They are actually very good with kids!', 'yeah but there are a lot of programs that help!']}, {'convai2': ['yum . i like to make lasagna and it s so good', 'yes ! trying to master lasagna .', 'it beats ramen noodles for sure ! do you have any hobbies ?', 'piercings are cool . i do not have any though .', "i don't know . whatever i want . maybe chicken", 'it would be a fashion statement . my dad would not like it .'], 'empathetic_dialogues': ['Cool. I love italian. Real italian.', "See. I'm not a great cook.", 'I love coffee, actually. I drink a few cups every morning!', 'Thats awesome i used to do the tattoos and ... During handling of the above exception, another exception occurred: ValueError Traceback (most recent call last) in 4 return tokenizer(examples["personas"], examples["context"], examples["previous_utterance"], examples["free_messages"], examples["guided_messages"], examples["suggestions"], examples["guided_chosen_suggestions"],) 5 ----> 6 dataset_train = dataset_train.map(lambda examples: tokenizer(examples["personas"], examples["context"], examples["previous_utterance"], examples["free_messages"], examples["guided_messages"], examples["suggestions"], truncation=True), batched=True) 7 dataset_validation = dataset_validation.map(lambda examples: tokenizer(examples["personas"], examples["context"], examples["previous_utterance"], examples["free_messages"], examples["guided_messages"], examples["suggestions"], truncation=True), batched=True) 8 15 frames /usr/local/lib/python3.7/dist-packages/transformers/utils/generic.py in _missing_(cls, value) 292 def _missing_(cls, value): 293 raise ValueError( --> 294 f"{value} is not a valid {cls. __name__ }, please select one of {list(cls._value2member_map_.keys())}" 295 ) 296 ```