# How do I reduce DistilBERT model size?
**URL:**
**Author:**  [@burrt](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fburrt)%5C
**Post date:** [March 25, 2021, 10:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-reduce-distilbert-model-size%2F5104%2F1 "2021-03-25T22:36:11Z")
Hi everyone,
I am recently start using huggingface’s transformer library and used BERT model to fit my data, after training on AWS sagemaker exported model is 300+ MB each. Then I tried distilBERT, it reduced to around 200MB, yet still too big to invoke if put into multi model endpoint. Is there anyway to reduce the size of distilBERT even more so I can fit them in the multi model endpoint?
---
**Author:**  [@OlivierCR](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOlivierCR)%5C
**Post date:** [April 1, 2021, 5:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-reduce-distilbert-model-size%2F5104%2F2 "2021-04-01T17:46:04Z")
Hi ! what makes it too big to invoke in multi-model endpoint? not enough time to deploy? invocation too slow?
---
**Author:**  [@jominmathew](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjominmathew)%5C
**Post date:** [April 2, 2021, 8:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-reduce-distilbert-model-size%2F5104%2F3 "2021-04-02T08:01:57Z")
Check Do we really need to distill? Jont Loss is all we need.
> **[GitHub - legacyai/tf-transformers: State of the art faster Transformer with...](https://github.com/legacyai/tf-transformers)**
>
> State of the art faster Transformer with Tensorflow 2.0 ( NLP, Computer Vision, Audio ). - GitHub - legacyai/tf-transformers: State of the art faster Transformer with Tensorflow 2.0 ( NLP, Computer...
This model is 4x smaller than DistillBERT and better than it by GLUE score of 4 points.
---
**Author:**  [@burrt](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fburrt)%5C
**Post date:** [April 6, 2021, 12:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-reduce-distilbert-model-size%2F5104%2F4 "2021-04-06T00:45:05Z")
It seems that after I put BERT models in a multi-model model and create an endpoint for it, when I tried to invoke the endpoint by targeting specific model, it will timeout. And BERT models are stored in S3 buckets. So I assume invocation is too slow.
---
**Author:**  [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [April 6, 2021, 9:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-reduce-distilbert-model-size%2F5104%2F5 "2021-04-06T21:13:26Z")
hey @burrt there’s a few threads on the forum about making transformer models smaller / faster, e.g.
- [Deploy multilingual sentence tansformer into cloud](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdeploy-multilingual-sentence-tansformer-into-cloud%2F4354)
- [Deploying inference model size and performance](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdeploying-inference-model-size-and-performance%2F3575)
my standard recommendation is to try quantization followed by ONNX / ONNX Runtime. before doing that though, i’d first try to understand what’s causing the timeout on your endpoint - it might be unrelated to the model and you don’t want to spend a lot of time optimising the wrong thing 🙂
---
**Author:**  [@OlivierCR](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FOlivierCR)%5C
**Post date:** [April 7, 2021, 6:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-reduce-distilbert-model-size%2F5104%2F6 "2021-04-07T18:44:11Z")
I agree with @lewtun i’d first diagnose what’s causing the timeout. You can add logs to your inference functions (model loading and inference) and check the `ModelLoadingWaitTime` SageMaker endpoint metric in CloudWatch to check what steps take time. If model loading is the issue, [according to the SageMaker doc](https://docs.aws.amazon.com/sagemaker/latest/dg/multi-model-endpoints.html#multi-model-caching) it may speed things up if you use a `d` type instance, that have local NVMe SSDs instead of using virtual EBS drives for storage. Also, try using bigger instances (in AWS, network bandwidth is generally proportional to instance size).
If using regular SageMaker endpoints is enough (and not SageMaker MME), you can take a look at that demo that deploys the bigger `bert-base-cased` on SageMaker [GitHub - aws-samples/amazon-sagemaker-bert-classify-pytorch: This sample show you how to train BERT on Amazon Sagemaker using Spot instances](https://github.com/aws-samples/amazon-sagemaker-bert-classify-pytorch)
---
**Author:**  [@burrt](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fburrt)%5C
**Post date:** [April 12, 2021, 9:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-do-i-reduce-distilbert-model-size%2F5104%2F7 "2021-04-12T21:11:41Z")
Thank you for your reply. SageMaker engpoints(not MME) is enough. After I deploy the endpoint with one single BERT model, it responds normally and can predict. After I deploy a multi-model endpoint with one or more BERT models in it, the cloudwatch just says model\_fn is not provided yet it’s in the script that is in the model artifacts.