# The fastest LLM inference on the server

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fthe-fastest-llm-inference-on-the-server%2F101461%3E%5C
**Category:** Research\
**Created:** [August 8, 2024, 8:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fthe-fastest-llm-inference-on-the-server%2F101461 "2024-08-08T08:17:08Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![GreyWolfBanana](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/greywolfbanana/32/30303_2.png) [@GreyWolfBanana](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FGreyWolfBanana)%5C
**Post date:** [August 8, 2024, 8:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fthe-fastest-llm-inference-on-the-server%2F101461%2F1 "2024-08-08T08:17:08Z")

</div>

Hello everyone! I am new to LLM rollout. I am using hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 model, but I can change it to a similar one if necessary. Please tell me what state of the art technologies exist now that will allow me to get the fastest inference, considering that I am rolling out the model on the server and it should respond to several people at once and do it quickly. I am running it on A100 80 GB. Currently I use vLLM to run with these parameters:

```auto
CUDA_VISIBLE_DEVICES=0 vllm serve hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 --quantization awq --tensor-parallel-size 1 --max-model-len 4096 --host 0.0.0.0 --port 8080 --rope-scaling='{"type": "dynamic", "factor": 8.0, "low_freq_factor": 1.0, "high_freq_factor": 4.0, "original_max_position_embeddings": 8192}'

```

With this approach, with a load of 1 request per second, each request generates 128 new tokens in 2 seconds.  
Maybe there are better ways? Maybe vLLM is launched somehow incorrectly? Maybe it is possible to somehow compile another model, but I do not know the necessary tools for this? I will be very glad if you help me.