# How to load mT5 checkpoint (.ckpt)?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-mt5-checkpoint-ckpt%2F3474%3E%5C
**Category:** Beginners\
**Created:** [February 2, 2021, 12:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-mt5-checkpoint-ckpt%2F3474 "2021-02-02T12:53:23Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Skylixia](https://avatars.discourse-cdn.com/v4/letter/s/439d5e/32.png) [@Skylixia](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSkylixia)%5C
**Post date:** [February 2, 2021, 12:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-mt5-checkpoint-ckpt%2F3474%2F1 "2021-02-02T12:53:23Z")

</div>

Hello,

I loaded mT5 with  
model = MT5ForConditionalGeneration.from\_pretrained(“google/mt5-small”)  
tokenizer = MT5Tokenizer.from\_pretrained(“google/mt5-small”)

Then, I fined-tuned it with Pytorch Lightning and now have a checkpoint in the format .ckpt. However, now when I try to load it again with

```
model = AutoModelWithLMHead.from_pretrained("path/to/checkpoint.ckpt")
tokenizer = MT5Tokenizer.from_pretrained("google/mt5-small")

```

it fails …  
I have this error code :

```
Traceback (most recent call last):
   line 8, in <module>
    model = AutoModelWithLMHead.from_pretrained("path/to/checkpoint.ckpt")
 "/transformers/models/auto/modeling_auto.py", line 930, in from_pretrained
    pretrained_model_name_or_path, return_unused_kwargs=True, **kwargs
  "/transformers/models/auto/configuration_auto.py", line 360, in from_pretrained
    config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  "/transformers/configuration_utils.py", line 423, in get_config_dict
    config_dict = cls._dict_from_json_file(resolved_config_file)
  "/transformers/configuration_utils.py", line 506, in _dict_from_json_file
    text = reader.read()
  File "/usr/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 64: invalid start byte

```

I have seen that it can be necessary to have a along with the checkpoint a config.json, vocab.json and merges.txt. The config.json is available on the google/mt5-small huggingface page so I downloaded it and added it to the folder with the checkpoint. However, the other files are not there. What are the steps for me to be able to load the .ckpt of mT5 ?

Thanks

---

<div class="post-metadata">

**Author:** ![milyiyo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/milyiyo/32/59140_2.png) [@milyiyo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmilyiyo)%5C
**Post date:** [February 1, 2022, 9:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-mt5-checkpoint-ckpt%2F3474%2F2 "2022-02-01T21:55:24Z")

</div>

Hi @Skylixia,

Did you manage to solve the problem?

---

<div class="post-metadata">

**Author:** ![lijingxin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lijingxin/32/3568_2.png) [@lijingxin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flijingxin)%5C
**Post date:** [March 17, 2022, 6:44am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-mt5-checkpoint-ckpt%2F3474%2F3 "2022-03-17T06:44:31Z")

</div>

Any updates？Maybe try to convert the ckpt to pytorch\_model.bin? but anybody know how to do this?