# Gpt2 model training , Loss nan

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-model-training-loss-nan%2F46242%3E%5C
**Category:** Intermediate\
**Created:** [July 10, 2023, 7:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-model-training-loss-nan%2F46242 "2023-07-10T07:00:43Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![irfan767](https://avatars.discourse-cdn.com/v4/letter/i/46a35a/32.png) [@irfan767](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Firfan767)%5C
**Post date:** [July 10, 2023, 7:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-model-training-loss-nan%2F46242%2F1 "2023-07-10T07:00:43Z")

</div>

> I am following [/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-gpt2-for-question-answering%2F31895%5D(https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tuning-gpt2-for-question-answering%2F31895) tutorial of huggingface library. Since I have a large dataset so I have used Dataloader and created batches. But am facing this error. Here is the process of creating batches,

class FeedbackEssentials(Dataset):  
def **init** (self, qa\_pairs, tokenizer, max\_length):  
self.qa\_pairs = qa\_pairs  
self.tokenizer = tokenizer  
self.max\_length = max\_length

def **len** (self):  
return len(self.qa\_pairs)

def **getitem** (self, idx):  
question = self.qa\_pairs[idx][0]  
text = f"{question} {self.tokenizer.eos\_token}"  
input\_ids = self.tokenizer.encode(text, add\_special\_tokens=True, max\_length=self.max\_length, padding=‘max\_length’, truncation=True)  
attention\_mask = [1] \* len(input\_ids) # Assuming all tokens should be attended to

```
return {
    'input_ids': torch.tensor(input_ids),
    'attention_mask': torch.tensor(attention_mask)
}

```

def text\_manipulation(train\_dataset):  
column1\_values = train\_dataset[‘Total Marks’].values  
column2\_values = train\_dataset[‘Coding’].values  
listOfLists = [[pair[0], pair[1]] for pair in zip(column1\_values, column2\_values)]

text = “”  
for feedback in listOfLists:  
text += f"{feedback[0]} {feedback[1]} {tokenizer.eos\_token}"  
return text

training\_dataset = text\_manipulation(dataset)  
max\_length\_training = max(len(tokenizer.encode(qa\_pair[0],add\_special\_tokens=True)) for qa\_pair in training\_dataset)  
dataset\_training = FeedbackEssentials(training\_dataset, tokenizer, max\_length\_training)  
batch\_size = 4  
dataloader = DataLoader(dataset\_training, batch\_size=batch\_size, shuffle=True)

optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5)  
scheduler = torch.optim.lr\_scheduler.StepLR(optimizer, step\_size=1, gamma=0.9)