# Cannot run large models using API token **URL:** **Author:** ![AndreaSottana](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@AndreaSottana](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAndreaSottana)%5C **Post date:** [February 16, 2023, 1:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcannot-run-large-models-using-api-token%2F31844%2F1 "2023-02-16T13:32:34Z") Hello, I am having the following two issues. 1. I cannot run large models using the inference API. For example, if I run the following, ```auto import requests API_URL = "/static-proxy?url=https%3A%2F%2Fapi-inference.huggingface.co%2Fmodels%2FEleutherAI%2Fgpt-neox-20b" headers = {"Authorization": "Bearer "} def query(payload): response = requests.post(API_URL, headers=headers, json=payload) return response.json() output = query({ "inputs": "Can you please let us know more details about your ", }) print(output) ``` I get this error ```auto {'error': 'Model EleutherAI/gpt-neox-20b is currently loading', 'estimated_time': 1651.7474365234375} ``` Why does this happen, and is there a way around the issue? 1. Even for the smaller models I do manage to run successfully, the output is different from the one generated in the user interface, for example the code below ```auto import requests API_URL = "/static-proxy?url=https%3A%2F%2Fapi-inference.huggingface.co%2Fmodels%2FEleutherAI%2Fgpt-neo-2.7B" headers = {"Authorization": "Bearer "} def query(payload): response = requests.post(API_URL, headers=headers, json=payload) return response.json() output = query({ "inputs": "Can you please let us know more details about your ", }) print(output) ``` generated the following ```auto [{'generated_text': 'Can you please let us know more details about your \nschedule.\n\nThanks, \nLiz Taylor \n\n-----Original Message----- \nFrom: Dasovich, Jeff [mailto:Jeff.D'}] ``` but on the website it is different (see below) ![Screenshot 2023-02-16 at 14.30.25](https://us1.discourse-cdn.com/hellohellohello/original/2X/c/cf9ab65688844f62d24e2b8ec6d5a9bf3ddc8eff.png) Why is this the case? Is there a way to ensure the outputs using the free Inference API are more aligned with those of the web UI? Thanks ---
**Author:** ![radames](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/radames/32/28246_2.png) [@radames](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fradames)%5C **Post date:** [February 16, 2023, 7:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcannot-run-large-models-using-api-token%2F31844%2F2 "2023-02-16T19:34:31Z")
> [@AndreaSottana](#): > > I cannot run large models using the inference API. For example, if I run the following, hi @AndreaSottana , that is a very large model, it takes a long time to load on our API inference. Our API inference is suitable for testing and evaluation. If you’re looking for less latency you probably need our dedicated service [Inference Endpoints](https://huggingface.co/inference-endpoints) You can read more about how the hub inference API works here > **[Overview](https://huggingface.co/docs/api-inference/quicktour#model-loading-and-latency)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. ---
**Author:** ![AndreaSottana](https://avatars.discourse-cdn.com/v4/letter/a/46a35a/32.png) [@AndreaSottana](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAndreaSottana)%5C **Post date:** [February 17, 2023, 10:22am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcannot-run-large-models-using-api-token%2F31844%2F3 "2023-02-17T10:22:11Z")
Hi @radames Many thanks for letting me know. I have two follow up questions: 1. Is there a way to load one of your large models locally for inference using model parallelism without having to manually edit the model’s internal code? I have access to multiple GPUs, and a single GPU with 24GB cannot load a 20 billion parameters model in memory, but with multiple GPUs it would be possible, however when I load the model it only tries to load it onto a single GPU and then gives an out of memory error. 2. Why are the results given by the UI when I run a model on the hub non-deterministic, but when I download it locally and run inference it appears to be deterministic and give a different output than the one I get on the hub? I am talking mostly about generative models. Thanks a lot ---
**Author:** ![pdeubel](https://avatars.discourse-cdn.com/v4/letter/p/45deac/32.png) [@pdeubel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpdeubel)%5C **Post date:** [March 8, 2023, 11:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcannot-run-large-models-using-api-token%2F31844%2F4 "2023-03-08T11:35:45Z")
I am also unable to run a large model on the inference API, specifically [Salesforce/codegen-16B-mono](https://huggingface.co/Salesforce/codegen-16B-mono). Neither trying it out using the widget on the website nor using a REST request through python works. In both cases I get a time out, for example the widget gives the following output after some time: `Model Salesforce/codegen-16B-mono time out`. Is that because the model is too big or because something in the backend is broken for that model. For the latter case, should I ask the model’s authors for help? ---
**Author:** ![mandelakori](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mandelakori/32/26960_2.png) [@mandelakori](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmandelakori)%5C **Post date:** [February 22, 2024, 2:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcannot-run-large-models-using-api-token%2F31844%2F5 "2024-02-22T14:38:53Z")
Hi, This makes sense however these two models: [zephyr](https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha) [aisak-assistant](https://huggingface.co/mandelakori/aisak-assistant) are the exact same size, but the former is able to run on inference api, wheras the latter cannot. Could you please provide me with a solution/explanation? Thanks ---
**Author:** ![radames](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/radames/32/28246_2.png) [@radames](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fradames)%5C **Post date:** [February 22, 2024, 11:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcannot-run-large-models-using-api-token%2F31844%2F6 "2024-02-22T23:26:35Z")
Hi @mandelakori, Zephyr is an LLM developed by our team, for which we’ve manually enabled inference. For other large models, we currently recommend using [Inference Endpoints](https://huggingface.co/inference-endpoints).