# Advice to speed and performance **URL:** **Author:** ![datistiquo](https://avatars.discourse-cdn.com/v4/letter/d/f0a364/32.png) [@datistiquo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdatistiquo)%5C **Post date:** [October 27, 2020, 4:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadvice-to-speed-and-performance%2F1769%2F1 "2020-10-27T16:48:39Z") Hey, I get the feeling that I might miss something about the perfomance and speed and memory issues using huggingface transformer. Since, I like this repo and huggingface transformers very much (!) I hope I do not miss something as I almost did not use any other Bert Implementations. Because I want to use TF2 that is why I use huggingface. Now, I would like to speed up inference and maybe decreasing memory usage. As I am native tensorflow user, I have no experience with the pytorch models at all. - So is it possible that the pytorch models are more performant and more efficient than the tf models? \*How can I speed up inference ? For encoding 200 sentences pairs on my cpu it takes 12 seconds. - So is it more feasible to use pytorch models for making inference or even training? Are there any memory usage differences? \*So, why is bert-as-a-service more performant and faster (as it looks like) I hope I can test this? I ask because I stumpled over here: > > > Hi, > > I wondered if anybody knows the difference between the \`BertModel\` of th…is repo and \[bert-as-service\](https://github.com/hanxiao/bert-as-service). > > 1. I cannot get the same result between these two even if I use the same checkpoint. pytorch-pretrained-BERT yield a lower acc and slower convergence. > > 2. The memory usage of \`BertModel\` seems much higher than bert-as-service. With the same batch-size=32, max\_seq\_len=100, the bert-as-service will take about 8000MB but \`BertModel\` will cost more than 16000MB because I got an OOM issue. > > Does any one knows the reason behind it? Is there any optimization done for bert-as-service? > [@How to ensure fast inference on both CPU and GPU with BertForSequenceClassification?](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-ensure-fast-inference-on-both-cpu-and-gpu-with-bertforsequenceclassification%2F1694)%3A > > Hi! I’d like to perform fast inference using BertForSequenceClassification on both CPUs and GPUs. For the purpose, I thought that torch DataLoaders could be useful, and indeed on GPU they are. Given a set of sentences sents I encode them and employ a DataLoader as in encoded\_data\_val = tokenizer.batch\_encode\_plus(sents, add\_special\_tokens=True, return\_attention\_mask=True, … Some advices for better usage (for deployment) are very appreciated. ---
**Author:** ![datistiquo](https://avatars.discourse-cdn.com/v4/letter/d/f0a364/32.png) [@datistiquo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdatistiquo)%5C **Post date:** [November 5, 2020, 1:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadvice-to-speed-and-performance%2F1769%2F2 "2020-11-05T13:56:00Z")
Is huggingface with pytorch faster than with tensorlfow? ---
**Author:** ![thomwolf](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thomwolf/32/2198_2.png) [@thomwolf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fthomwolf)%5C **Post date:** [November 12, 2020, 1:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadvice-to-speed-and-performance%2F1769%2F3 "2020-11-12T13:15:26Z")
@jplu is currently working on making the TF2 models a lot faster! The situation should be better soon (still in a few weeks probably). ---
**Author:** ![jplu](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jplu/32/148_2.png) [@jplu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjplu)%5C **Post date:** [November 13, 2020, 9:34am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadvice-to-speed-and-performance%2F1769%2F4 "2020-11-13T09:34:20Z")
Hello ! As thomwolf said, we are currently working on a much performant version of the TF models and then for now, yes, the PyTorch models are more optimized than the TF ones. Bert-as-a-service works faster because it is highly optimized for inference by making: - BERT using mixed precision (thing that we can do as well) - freeze the model - powerful/scalable service API If you are looking for a performant inference for your TF model I suggest you to take a look at ONNX, we provide a [script](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_graph_to_onnx.py) to create your own optimized ONNX model in the repo. And afterwards you can run a [Triton server](https://github.com/triton-inference-server/server) to provide an over your model. ---
**Author:** ![datistiquo](https://avatars.discourse-cdn.com/v4/letter/d/f0a364/32.png) [@datistiquo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdatistiquo)%5C **Post date:** [December 7, 2020, 9:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadvice-to-speed-and-performance%2F1769%2F5 "2020-12-07T21:52:20Z")
Hey, thank you. I think I could also use a finetuned TF model (with huggingface) and use it with bert-as-aservice?