# Can't save my finetuned model

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-save-my-finetuned-model%2F116472%3E%5C
**Category:** Beginners\
**Created:** [November 9, 2024, 4:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-save-my-finetuned-model%2F116472 "2024-11-09T04:00:53Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![changminbark](https://avatars.discourse-cdn.com/v4/letter/c/dfb087/32.png) [@changminbark](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fchangminbark)%5C
**Post date:** [November 9, 2024, 4:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-save-my-finetuned-model%2F116472%2F1 "2024-11-09T04:00:53Z")

</div>

Hi everyone, I have the following code below but it’s not saving the model. Can anyone give me suggestions or tell me why it’s not saving it?

```auto
import torch
from torch.utils.data import DataLoader
from torch.optim import AdamW
import pandas as pd
from app.utils import data_utils
from datasets import Dataset, load_dataset
from transformers import AutoTokenizer, DistilBertForSequenceClassification, get_scheduler
from tqdm.auto import tqdm
import evaluate

# Fine-tuning DistilBERT for news sentiment analysis (using Kaggle news sentiment analysis) -> 3 labels
# 0 = Negative
# 1 = Neutral
# 2 = Positive
model_name = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = DistilBertForSequenceClassification.from_pretrained(model_name, num_labels=3)

# Other test news sentiment
# ds = load_dataset("sara-nabhani/ML-news-sentiment")

# Tokenize Function
def tokenize_function(data):
    return tokenizer(data["text"], padding="max_length", truncation=True)

# https://www.kaggle.com/datasets/clovisdalmolinvieira/news-sentiment-analysis?resource=download
# dataset = load_dataset("yelp_review_full")
# print(dataset)

# Load in dataset and shape it into pandas dataframe
df = data_utils.load_raw_data("kaggle_news_sentiment_analysis.csv")
df["text"] = df["Title"] + ": " + df["Description"]
df = df[["text", "Sentiment"]]
replacements = {"negative": 0, "neutral": 1, "positive": 2}
df["Sentiment"] = df["Sentiment"].map(replacements).fillna(df["Sentiment"])
df.rename(columns={"Sentiment": "labels"}, inplace=True)
print(df)

# Create train and test datasets
train = df.head(3000)
test = df.tail(500)

# Convert from pandas dataframe to huggingface dataframe
train = Dataset.from_pandas(train)
print(train)
test = Dataset.from_pandas(test)
print(test)

# Tokenize the datasets
tokenized_train = train.map(tokenize_function, batched=True)
tokenized_train = tokenized_train.remove_columns(["text"])
tokenized_train.set_format(type="torch")
print(tokenized_train)

tokenized_test = test.map(tokenize_function, batched=True)
tokenized_test = tokenized_test.remove_columns(["text"])
tokenized_test.set_format(type="torch")
print(tokenized_test)

# Create data loaders
train_dataloader = DataLoader(tokenized_train, shuffle=True, batch_size=10)
test_dataloader = DataLoader(tokenized_test, batch_size=10)

# Initialize optimizer
optimizer = AdamW(model.parameters(), lr=5e-5)

# Initialize learning rate scheduler
num_epochs = 3
num_training_steps = num_epochs * len(train_dataloader)
lr_scheduler = get_scheduler(name="linear", optimizer=optimizer, num_warmup_steps=0, num_training_steps=num_training_steps)

# Specify device to train on
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
model.to(device)

# Training loop
print("STARTING TRAINING LOOP ==============")
progress_bar = tqdm(range(num_training_steps))

model.train()
for epoch in range(num_epochs):
    for batch in train_dataloader:
        batch = {k: v.to(device) for k, v in batch.items()}
        outputs = model(**batch) # Pass batch through model and get output
        loss = outputs.loss # Calculate loss from output
        loss.backward() # Backpropagation using losses to calculate gradient

        optimizer.step() # Update parameters
        lr_scheduler.step() #
        optimizer.zero_grad() # Refresh gradient
        progress_bar.update(1) # Update progress bar

# Save model
model.save_pretrained(save_directory="./saved_models/distilbert")

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [November 9, 2024, 4:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-save-my-finetuned-model%2F116472%2F2 "2024-11-09T04:05:10Z")

</div>

This may fix the problem. If it doesn’t, there may be a bug in one of the libraries.

```py
# Save model
model.to("cpu") # Added
model.save_pretrained(save_directory="./saved_models/distilbert")

```

> <https://github.com/huggingface/transformers/issues/29608>
>
> \### System Info
> 
> Transformers == 4.38.2
> Platform == TPU V4 on GKE
> Python == 3.…10
> 
> \### Who can help?
> 
> @ArthurZucker 
> 
> \### Information
> 
> \- \[\] The official example scripts
> \- \[X\] My own modified scripts
> 
> \### Tasks
> 
> \- \[\] An officially supported task in the \`examples\` folder (such as GLUE/SQuAD, ...)
> \- \[x\] My own task or dataset (give details below)
> 
> \### Reproduction
> 
> I ran some tests on a GKE Cluster with TPU V4 with 4 nodes.
> 
> https://gist.github.com/moficodes/1492228c80a3c08747a973b519cc7cda
> 
> This run fails with
> 
> Traceback (most recent call last):
> File "/usr/local/lib/python3.10/site-packages/safetensors/torch.py", line 13, in storage\_ptr
> return tensor.untyped\_storage().data\_ptr()
> RuntimeError: Attempted to access the data pointer on an invalid python storage.
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
> File "//fsdp.py", line 112, in \<module\>
> model.save\_pretrained(new\_model\_id)
> File "/usr/local/lib/python3.10/site-packages/transformers/modeling\_utils.py", line 2448, in save\_pretrained
> safe\_save\_file(shard, os.path.join(save\_directory, shard\_file), metadata={"format": "pt"})
> File "/usr/local/lib/python3.10/site-packages/safetensors/torch.py", line 281, in save\_file
> serialize\_file(\_flatten(tensors), filename, metadata=metadata)
> File "/usr/local/lib/python3.10/site-packages/safetensors/torch.py", line 470, in \_flatten
> shared\_pointers = \_find\_shared\_tensors(tensors)
> File "/usr/local/lib/python3.10/site-packages/safetensors/torch.py", line 72, in \_find\_shared\_tensors
> if v.device != torch.device("meta") and storage\_ptr(v) != 0 and storage\_size(v) != 0:
> File "/usr/local/lib/python3.10/site-packages/safetensors/torch.py", line 17, in storage\_ptr
> return tensor.storage().data\_ptr()
> File "/usr/local/lib/python3.10/site-packages/torch/storage.py", line 956, in data\_ptr
> return self.\_data\_ptr()
> File "/usr/local/lib/python3.10/site-packages/torch/storage.py", line 960, in \_data\_ptr
> return self.\_untyped\_storage.data\_ptr()
> RuntimeError: Attempted to access the data pointer on an invalid python storage.
> 
> \### Expected behavior
> 
> Save the model and push to hugging face.

---

<div class="post-metadata">

**Author:** ![changminbark](https://avatars.discourse-cdn.com/v4/letter/c/dfb087/32.png) [@changminbark](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fchangminbark)%5C
**Post date:** [November 9, 2024, 5:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-save-my-finetuned-model%2F116472%2F3 "2024-11-09T05:21:32Z")

</div>

I’ll try it out after this, but I was wondering if it was possible to train the model with native pytorch and still save it using HuggingFace’s save\_pretained() function. Would that also be another possible issue? I currently don’t get any error messages; it just doesn’t save it.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [November 9, 2024, 5:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-save-my-finetuned-model%2F116472%2F4 "2024-11-09T05:29:42Z")

</div>

Usually, if save\_pretrained fails, an error message is displayed in many cases, so it smells like a bug…  
Basically, what is saved with HF’s save\_pretrained is the torch model’s state\_dict and the configuration json, so converting a model trained with native torch to HF format is usually not that difficult.  
The only problem is when the state\_dict keys change.

> <https://github.com/pytorch/torchtune/issues/832>
>
> I'm finding this repo to be a user friendly, extensible, memory efficient soluti…on for training/fine-tuning models. However, when it comes to inference, there is a usability gap that could be solved by converting the model into a format that can be loaded by HF's \[\`from\_pretrained()\`\](https://huggingface.co/docs/transformers/v4.40.0/en/model\_doc/auto#transformers.AutoModel.from\_pretrained) function.
> 
> The specific thing I want to do is load a model fine-tuned with torchtune into a \[Gradio chatbot, complete with token streaming\](https://www.gradio.app/guides/creating-a-chatbot-fast#example-using-a-local-open-source-llm-with-hugging-face). I imagine many other downstream tasks would be made easier with this functionality as well.
> 
> Would it be reasonable to add the following options to the checkpointer?
> 
> \- Save full model weights in such a way that they can be loaded with \[\`from\_pretrained()\`\](https://huggingface.co/docs/transformers/v4.40.0/en/model\_doc/auto#transformers.AutoModel.from\_pretrained).
> \- Save a LoRA adapter in such a way that it can be loaded into a model with \[\`get\_peft\_model()\`\](https://huggingface.co/docs/peft/v0.10.0/en/package\_reference/peft\_model#peft.get\_peft\_model).
> 
> If this seems like a valid addition, and isn't a huge lift, I would be happy to give it a try.

> **[Serialization](https://huggingface.co/docs/huggingface_hub/v0.26.2/package_reference/serialization)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![changminbark](https://avatars.discourse-cdn.com/v4/letter/c/dfb087/32.png) [@changminbark](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fchangminbark)%5C
**Post date:** [November 9, 2024, 4:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-save-my-finetuned-model%2F116472%2F5 "2024-11-09T16:25:30Z")

</div>

I found out the problem. I forgot to put return\_tensors=“pt”. 🥲 😭 😭

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [November 10, 2024, 4:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcant-save-my-finetuned-model%2F116472%2F6 "2024-11-10T04:25:36Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.