# How to convert TF Checkpoints to sentence embedings
**URL:**
**Author:**  [@peterpirog](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpeterpirog)%5C
**Post date:** [November 11, 2020, 10:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-convert-tf-checkpoints-to-sentence-embedings%2F2004%2F1 "2020-11-11T22:07:07Z")
Hello
I trained the model with original bert code [https://github.com/google-research/bert](https://github.com/google-research/bert) but I don’t know:
1. How to convert files to the model: 
I tried to understand the documentation but deinetly i do something wrong ( I don’t know what I have to do with index and meta file, why file model.ckpt-500000.data-00000-of-00001 is much bigger than typical model.ckpt file for Bert standard, why I don’t have model.ckpt file as aoutput?)
[https://huggingface.co/transformers/main\_classes/model.html#transformers.PreTrainedModel](https://huggingface.co/transformers/main_classes/model.html#transformers.PreTrainedModel)
1. How to convert whole sentence to embeding vector? ( I know how to load tokenizer from file vocab.txt)
I’am afraid that I didn’t understand some general concept of transformers models and this is a reason of my problems. Are there any online course for transformers library?
Regards Peter
---
**Author:**  [@thomwolf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fthomwolf)%5C
**Post date:** [November 12, 2020, 1:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-convert-tf-checkpoints-to-sentence-embedings%2F2004%2F2 "2020-11-12T13:04:40Z")
To convert a model trained with the original repository you should use the conversion script here: [https://github.com/huggingface/transformers/blob/master/src/transformers/convert\_bert\_original\_tf\_checkpoint\_to\_pytorch.py](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py)
---
**Author:**  [@peterpirog](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpeterpirog)%5C
**Post date:** [November 12, 2020, 3:53pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-convert-tf-checkpoints-to-sentence-embedings%2F2004%2F3 "2020-11-12T15:53:28Z")
Thank You. I tried code below:
and result was the error:
" File “C:\ProgramData\Anaconda3\lib\site-packages\tensorflow\python\training\py\_checkpoint\_reader.py”, line 44, in error\_translator
raise errors\_impl.DataLossError(None, None, error\_message)
tensorflow.python.framework.errors\_impl.DataLossError: Unable to open table file G:\PycharmProject\Ancient\_Greek\_BERT\model\_uncased\_L-12\_H-512\_A-12\model.ckpt-500000.data-00000-of-00001: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
"
I have no idea how to modify my code, because error description isn’t clear for me.
Peter
---
**Author:**  [@peterpirog](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpeterpirog)%5C
**Post date:** [November 12, 2020, 10:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-convert-tf-checkpoints-to-sentence-embedings%2F2004%2F4 "2020-11-12T22:14:59Z")
I found solution 🙂 It’s working.
---
**Author:**  [@flummox](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fflummox)%5C
**Post date:** [November 27, 2020, 6:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-convert-tf-checkpoints-to-sentence-embedings%2F2004%2F5 "2020-11-27T18:33:38Z")
I’m completely new to Transformers and BERT, and I’m struggling to pull together a functional workflow.
Like the OP, I’m trying to use convert\_pytorch\_checkpoint\_to\_tf2.py to convert a pre-trained model from Google Tensorflow BERT format (via TPU) into the pytorch version. I want to go on to fine-tune this model.
But, crucially, my pre-trained model is based on a custom vocabulary - I trained BERT from scratch on my own data. It isn’t like any of the Huggingface pre-trained models.
But I’m trying the Huggingface conversion script as below:
python convert\_pytorch\_checkpoint\_to\_tf2.py --model\_type=bert --pytorch\_checkpoint\_path=model.ckpt-20 --config\_file=bert\_config.json --tf\_dump\_path=pytorch\_model.bin
(The checkpoint is a ‘hello world’ example to help me understand the training workflow).
But I get an error from the conversion script:
File “convert\_pytorch\_checkpoint\_to\_tf2.py”, line 402, in convert\_all\_pt\_checkpoints\_to\_tf
config\_class, model\_class, pt\_model\_class, aws\_model\_maps, aws\_config\_map = MODEL\_CLASSES[model\_type]
ValueError: not enough values to unpack (expected 5, got 4)
When I put a diagnostic print(MODEL\_CLASSES[model\_type]) in the script, I see that the model\_type I gave (‘bert’) is being matched against many existing Huggingface pre-trained models. The simple text match finds the string ‘bert’ in the MODEL\_CLASSES list but it’s not fully matched against any Huggingface pre-trained model. So I think now that I must specify a Huggingface pre-trained model definition - but how is this used? As some sort of template for further training?
I am obviously missing the point totally here. What I want is to be able to pre-train a model from scratch with Tensorflow on Google TPU, or the equivalent with pytorch, then fine-tune it. Is that possible in the Huggingface ecosystem? Is there a HF script to do the conversion to pytorch format, without the constraint of specifying a pre-trained model? I think maybe I’m at saturation point, continually hitting dependency mismatches etc. as I try to learn to work with BERT.
I’d be very grateful for your advice.