# Optimize response time of model output

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Foptimize-response-time-of-model-output%2F13064%3E%5C
**Category:** 🤗Transformers\
**Created:** [December 23, 2021, 5:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Foptimize-response-time-of-model-output%2F13064 "2021-12-23T05:06:25Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![devpy079](https://avatars.discourse-cdn.com/v4/letter/d/ee7513/32.png) [@devpy079](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdevpy079)%5C
**Post date:** [December 23, 2021, 5:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Foptimize-response-time-of-model-output%2F13064%2F1 "2021-12-23T05:06:25Z")

</div>

I am using Parrot Paraphrase that uses transformers model, but the problem is that the result generation is too slow for it without GPU. I have limitation of GPU and can only process the result on CPU. I saw some optimization solutions but they are all suggesting to train the model in some desired form or optmization but they model is already trained by someone else so I can not do much on that. Recently I improved the performance of one of the model by pytorch quantize implementation. eg:

```auto
model = (AutoModelForSequenceClassification
         .from_pretrained(model_ckpt).to("cpu"))

model_quantized = quantize_dynamic(model, {nn.Linear}, dtype=torch.qint8)

```

Can I use it with the parrot library, the code it uses for the model loading is:

```auto
  self.tokenizer = AutoTokenizer.from_pretrained(model_tag)
   self.model = AutoModelForSeq2SeqLM.from_pretrained(model_tag)

```