Instructions to use HuggingFaceM4/Idefics3-8B-Llama3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use HuggingFaceM4/Idefics3-8B-Llama3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="HuggingFaceM4/Idefics3-8B-Llama3")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("HuggingFaceM4/Idefics3-8B-Llama3")
model = AutoModelForImageTextToText.from_pretrained("HuggingFaceM4/Idefics3-8B-Llama3")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use HuggingFaceM4/Idefics3-8B-Llama3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "HuggingFaceM4/Idefics3-8B-Llama3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceM4/Idefics3-8B-Llama3",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/HuggingFaceM4/Idefics3-8B-Llama3

SGLang

How to use HuggingFaceM4/Idefics3-8B-Llama3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "HuggingFaceM4/Idefics3-8B-Llama3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceM4/Idefics3-8B-Llama3",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "HuggingFaceM4/Idefics3-8B-Llama3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HuggingFaceM4/Idefics3-8B-Llama3",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use HuggingFaceM4/Idefics3-8B-Llama3 with Docker Model Runner:
```
docker model run hf.co/HuggingFaceM4/Idefics3-8B-Llama3
```

How to Effectively Run the Idefics 3 Model on AWS SageMaker for Inference

#15

by Mehyaar - opened Sep 5, 2024

Discussion

Mehyaar

Sep 5, 2024

•

edited Sep 5, 2024

Title: How to Effectively Run the Idefics 3 Model on AWS SageMaker for Inference on 19k Images

Hi everyone,

I’m currently working on a project where I need to run the Idefics 3 model for inference on a dataset of 19,000 images. I plan to use AWS SageMaker for this task.

Could anyone provide guidance on the following:

Configuration: What are the best practices for configuring AWS SageMaker to handle such a large inference task efficiently?
Instance Selection: Are there specific instance types or configurations that would be particularly suitable for running the Idefics 3 model on a dataset of this size?
Performance Optimization: Any tips or considerations for optimizing performance and managing costs during this process?
Integration: Are there any specific steps or scripts required to integrate and run the Idefics 3 model smoothly on SageMaker?

Any insights or experiences you can share would be incredibly helpful!

Thank you in advance!

Best,
Mehyar

Mehyaar changed discussion status to closed Sep 5, 2024

Mehyaar changed discussion status to open Sep 5, 2024

HugoLaurencon

Sep 5, 2024

The most important parameter for you is size= {"longest_edge": N*364} (detailed more in the model card) to choose the number of tokens you'll use for each image, which influences the efficiency at inference.