# HuggingFace Infernece Endpoint don't work
**URL:**
**Author:**  [@rashmika-d2167](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frashmika-d2167)%5C
**Post date:** [July 12, 2025, 3:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhuggingface-infernece-endpoint-dont-work%2F162765%2F1 "2025-07-12T15:18:43Z")
import os
from langchain\_huggingface import HuggingFaceEndpoint
from langchain.chains import LLMChain
from langchain\_core.prompts import PromptTemplate
from langgraph.checkpoint.memory import MemorySaver
from langgraph.graph import START, MessagesState, StateGraph
from langchain\_core.messages import SystemMessage, HumanMessage, AIMessage
from typing import TypedDict
from PIL import Image
from dotenv import load\_dotenv
load\_dotenv()
# HuggingFace API TOKEN
HUGGINGFACEHUB\_API\_TOKEN = os.getenv(“HUGGINGFACEHUB\_API\_TOKEN”)
os.environ[“HUGGINGFACEHUB\_API\_TOKEN”] = HUGGINGFACEHUB\_API\_TOKEN
endpoint\_url = ‘’
class HarmonyState(TypedDict):
messages: list
image: Image.Image
people: dict
sys\_msg = “”"
```
"""
```
template = “”"
```
"""
```
prompt = PromptTemplate.from\_template(template)
model = HuggingFaceEndpoint(
endpoint\_url=endpoint\_url,
max\_new\_tokens=512,
top\_k=15,
top\_p=0.95,
typical\_p=0.95,
temperature=0.1,
repetition\_penalty=1.03,
)
def call\_model(state: HarmonyState) → dict:
image = state[“image”]
conversation = “\n”.join([msg.content for msg in state[“messages”] if isinstance(msg, HumanMessage)])
prompt\_text = prompt.format(
conversation=conversation,
people=str(state[“people”]),
)
```
# Fix: Correct the model invocation format
response = model.invoke(prompt_text)
return {
"messages": state["messages"] + [AIMessage(content=response)],
"image": image,
"people": state["people"],
}
```
workflow = StateGraph(state\_schema=HarmonyState)
workflow.add\_node(“model”, call\_model)
workflow.set\_entry\_point(“model”)
workflow.set\_finish\_point(“model”)
memory = MemorySaver()
app = workflow.compile(checkpointer=memory)
def output\_of\_model(conversation, people):
image = Image.open(“output\_image.jpg”).convert(“RGB”)
initial\_state = {
“messages”: [
SystemMessage(content=sys\_msg),
HumanMessage(content=conversation)
],
“image”: image,
“people”: people
}
```
try:
# Fix: Use the correct method and configuration format
config = {"configurable": {"thread_id": "session-1"}}
result = app.invoke(initial_state, config=config)
print(result["messages"][-1].content)
return result["messages"][-1].content
except Exception as e:
print(f"Oops, something went wrong: {str(e)}")
return f"Oops, something went wrong: {str(e)}"
```
I try to create a bot using Hugging Face endpoint. Model I am using is qwen2-5-vl-7b-instruct-gguf-qzv with Llama.cpp Container. But every time I run the code 404 Error. I tried every possible way to solve it. I code this looking at LangChain document.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [July 12, 2025, 11:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhuggingface-infernece-endpoint-dont-work%2F162765%2F2 "2025-07-12T23:43:18Z")
> Model I am using is qwen2-5-vl-7b-instruct-gguf-qzv with Llama.cpp Container
This caught my attention. While it is easy to run LLM in Llama.cpp, running VLM is likely to be quite challenging.
In the case of TGI (Hugging Face Endpoint), using a backend such as vLLM would be a more reliable option with VLM.
>
>
> \### Prerequisites
>
> \- \[x\] I am running the latest code. Mention the version if po…ssible as well.
> \- \[x\] I carefully followed the \[README.md\](https://github.com/ggerganov/llama.cpp/blob/master/README.md).
> \- \[x\] I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
> \- \[x\] I reviewed the \[Discussions\](https://github.com/ggerganov/llama.cpp/discussions), and have a new and useful enhancement to share.
>
> \### Feature Description
>
> Is anybody implementing this?
>
> If not, I may give it a go. But it will take some time as I am new to the source side of llama.cpp/ggml.
>
>
>
> \### Motivation
>
> Well, it's not currently working. :-)
>
> \### Possible Implementation
>
> Based on the existing Qwen 2 VL implementation.
---
**Author:**  [@rashmika-d2167](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frashmika-d2167)%5C
**Post date:** [July 13, 2025, 12:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhuggingface-infernece-endpoint-dont-work%2F162765%2F3 "2025-07-13T00:28:41Z")
Thank you for your reply, I will give a try