# How to use set\_transform when map becomes unfeasible?
**URL:**
**Author:**  [@pamo77](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpamo77)%5C
**Post date:** [June 16, 2024, 12:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-set-transform-when-map-becomes-unfeasible%2F92415%2F1 "2024-06-16T12:20:49Z")
Hello,
For context: I am working on a sequence classification task, using a RoBERTa derivative model
```auto
from transformers import AutoModelForSequenceClassification
model_Mol_seq2seq = AutoModelForSequenceClassification.from_pretrained(“model/name”, num_labels=8, deterministic_eval=True, trust_remote_code=True)
tokenizer_3 = AutoTokenizer.from_pretrained(“model/name”, trust_remote_code=True)
```
Furthermore, I have a large dataset with two columns labelled `text` and `label`, respectively, and I need to tokenise the values in the `text` column.
The tokeniser is wrapped like this:
```auto
def tokenize_function(examples, col='text'):
return tokenizer(examples[col], truncation=True, padding='max_length', max_length=768)
```
If I use the `map` method and apply the tokeniser on the `text` column the size of the dataset explodes to several hundreds of GB, which I cannot find storage for. Therefore, I want to use `.set_transform(tokenize_function)` on the dataset.
When using the map method with the tokeniser\_function on the data set (i.e. dataset = dataset.map(tokenizer\_function)), I get a dataset with the following columns:
```auto
Dataset({
features: ['text', 'label', 'input_ids', 'attention_mask'],
num_rows: 1
})
```
which is what I expect.
However, if I use `dataset.set_transform(tokenizer_function)`, `dataset` yields
```auto
(Dataset({
features: ['text', 'label'],
num_rows: 1
})
```
and `dataset[0] = {'input_ids': [0,4, 9, …], attention_mask': [1,1,1,….]}`, and there is no entry for `label`. When using the `dataset.set_transform(…)` option as input in `Trainer`, I get the following error:
`ValueError: The model did not return a loss from the inputs, only the following keys: logits. For reference, the inputs it received are input_ids,attention_mask.`
What am I doing wrong?
Many thanks in advance.
PS! I know similar questions have been asked, but couldn’t find a recent one that addresses the above, specifically.
---
**Author:**  [@pamo77](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpamo77)%5C
**Post date:** [June 19, 2024, 9:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-set-transform-when-map-becomes-unfeasible%2F92415%2F2 "2024-06-19T21:46:34Z")
For what it is worth, copilot suggested the following (and now rather obvious) solution: to modify the `tokenize_function` function as follows:
```auto
def tokenize_function(examples, col=col_in, max_length=768):
tokenized_inputs = tokenizer(examples[col], truncation=True, padding='max_length', max_length=max_length)
# Include the labels
tokenized_inputs["labels"] = examples["label"]
return tokenized_inputs
```
which allowed me to progress.
---
**Author:**  [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [June 20, 2024, 7:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-use-set-transform-when-map-becomes-unfeasible%2F92415%2F3 "2024-06-20T19:51:36Z")
Closed on June 20, 2024, 7:51pm UTC
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.