# Integration and Scale

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fintegration-and-scale%2F106069%3E%5C
**Category:** Inference Endpoints on the Hub\
**Created:** [September 9, 2024, 8:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fintegration-and-scale%2F106069 "2024-09-09T20:21:19Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![grantdozier](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@grantdozier](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgrantdozier)%5C
**Post date:** [September 9, 2024, 8:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fintegration-and-scale%2F106069%2F1 "2024-09-09T20:21:19Z")

</div>

My name is Grant, and I’m reaching out regarding the integration of a Hugging Face inference endpoint into our environment.

We are specifically considering the following endpoint for example: [https://ui.endpoints.huggingface.co/grantdozier/new?repository=bigscience%2Fbloom](https://ui.endpoints.huggingface.co/grantdozier/new?repository=bigscience%2Fbloom)

Our primary question concerns the capacity of a single dedicated endpoint. Specifically:

**1. How many concurrent users can one dedicated endpoint serve?**  
**2. What is the recommended ratio of endpoints to users for optimal performance?**

This information is crucial for our planning, as we anticipate scaling to approximately 1,000 users within the next two months. Our goal is to determine the most efficient number of dedicated endpoints needed to support our user base.

Additionally, we are evaluating several models on your platform for potential dedicated endpoint deployment. Any insights you can provide on best practices for scaling and performance optimization would be greatly appreciated.

---

<div class="post-metadata">

**Author:** ![mahmutc](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mahmutc/32/64626_2.png) [@mahmutc](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmahmutc)%5C
**Post date:** [September 10, 2024, 12:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fintegration-and-scale%2F106069%2F2 "2024-09-10T12:22:37Z")

</div>

hi @grantdozier

It depends on your configuration and user.

huggingface suggests Nvidia A100 (8 GPUs · 640 GB - 88 vCPUs · 1160 GB) for **bigscience/bloom** but of course you can try smaller instances first.

You might be interested in this article:

> **[Why we’re switching to Hugging Face Inference Endpoints, and maybe you should...](https://medium.com/mantisnlp/why-were-switching-to-hugging-face-inference-endpoints-and-maybe-you-should-too-829371dcd330)**
>
> Hugging Face recently launched Inference Endpoints. In this blog post we look at some of the pros and cons, and why we’re using them.

And this one is about scaling:

> **[Autoscaling](https://huggingface.co/docs/inference-endpoints/autoscaling)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![grantdozier](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@grantdozier](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgrantdozier)%5C
**Post date:** [September 11, 2024, 8:12pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fintegration-and-scale%2F106069%2F3 "2024-09-11T20:12:46Z")

</div>

Thank you so much for taking the time to respond. HuggingFace support is really awesome! I liked the article about "Why we’re switching … ". I found the aspect about having a multi-model endpoint extremely interesting. We are looking to have about 5 different models, can you tell me a bit about how we would go about implementing this, or point us to an example of this being implemented successfully?