Instructions to use wavespeed/MiniCPM-V-4_5-abliterated-int8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use wavespeed/MiniCPM-V-4_5-abliterated-int8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="wavespeed/MiniCPM-V-4_5-abliterated-int8", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("wavespeed/MiniCPM-V-4_5-abliterated-int8", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use wavespeed/MiniCPM-V-4_5-abliterated-int8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "wavespeed/MiniCPM-V-4_5-abliterated-int8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wavespeed/MiniCPM-V-4_5-abliterated-int8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/wavespeed/MiniCPM-V-4_5-abliterated-int8

SGLang

How to use wavespeed/MiniCPM-V-4_5-abliterated-int8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "wavespeed/MiniCPM-V-4_5-abliterated-int8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wavespeed/MiniCPM-V-4_5-abliterated-int8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "wavespeed/MiniCPM-V-4_5-abliterated-int8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wavespeed/MiniCPM-V-4_5-abliterated-int8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use wavespeed/MiniCPM-V-4_5-abliterated-int8 with Docker Model Runner:
```
docker model run hf.co/wavespeed/MiniCPM-V-4_5-abliterated-int8
```

About this repository

An 8-bit (bitsandbytes LLM.int8()) quantization of huihui-ai/Huihui-MiniCPM-V-4_5-abliterated, published by WaveSpeed AI.

Note what the base model is: abliterated means the upstream author ablated the refusal direction out of MiniCPM-V-4.5. This model will therefore answer prompts the original declines, and it is on you to put your own safety layer in front of it. If you want the original behaviour, quantize openbmb/MiniCPM-V-4_5 instead.

MiniCPM-V-4.5-abliterated-int8

This is an 8-bit quantized version of huihui-ai/Huihui-MiniCPM-V-4_5-abliterated using bitsandbytes int8 quantization.

Model Details

Base Model: huihui-ai/Huihui-MiniCPM-V-4_5-abliterated
Quantization: 8-bit integer using bitsandbytes
Model Size: ~9.35 GB (79.4% reduction from original 45.28 GB)
Compute dtype: float16
Quantization method: LLM.int8() with mixed-precision decomposition

Quantization Configuration

{
  "load_in_8bit": true,
  "bnb_8bit_compute_dtype": "float16",
  "bnb_8bit_quant_type": "int8",
  "llm_int8_skip_modules": ["lm_head", "vision"],
  "llm_int8_threshold": 6.0,
  "quant_method": "bitsandbytes"
}

Key Features

Mixed Precision: Uses int8 for weights with fp16 for activations
Outlier Management: Automatically handles outliers in fp16 for better accuracy
Selective Quantization: Skips critical modules (lm_head, vision) to preserve quality
Better accuracy than int4: While larger than 4-bit, provides significantly better quality

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "wavespeed/MiniCPM-V-4_5-abliterated-int8",
    device_map="auto",
    trust_remote_code=True,
    torch_dtype=torch.float16
)

tokenizer = AutoTokenizer.from_pretrained(
    "wavespeed/MiniCPM-V-4_5-abliterated-int8",
    trust_remote_code=True
)

# For inference
# The model will automatically use int8 weights with fp16 compute

Requirements

transformers>=4.35.0
bitsandbytes>=0.41.0
torch>=2.0.0
accelerate>=0.20.0
CUDA-capable GPU (int8 quantization requires CUDA)

Performance Notes

Memory Usage: ~9.35 GB VRAM required
Speed: Slightly slower than fp16 due to dequantization overhead
Quality: Better preservation of model quality compared to 4-bit quantization
Best for: Users who need better quality than 4-bit but still want memory savings

Comparison with Other Quantizations

Version	Size	Relative Quality	Use Case
Original (fp16)	45.28 GB	Best	Maximum quality, high VRAM
int8 (this)	9.35 GB	Very Good	Balanced quality/memory
int4	6.09 GB	Good	Maximum memory savings

License

Same as the original model - please refer to the base model's license.

Acknowledgments

Original model by huihui-ai
Quantization using bitsandbytes LLM.int8() method

Downloads last month: 20

Safetensors

Model size

9B params

Tensor type

F32

F16

Model tree for wavespeed/MiniCPM-V-4_5-abliterated-int8

Base model

openbmb/MiniCPM-V-4_5

Quantized

huihui-ai/Huihui-MiniCPM-V-4_5-abliterated

Quantized

(3)

this model

Collection including wavespeed/MiniCPM-V-4_5-abliterated-int8

Quantized weights we publish

Collection

FP8, INT8, NF4 and precision-cast variants of open models. Each card says exactly which tensors changed and which were left alone. • 16 items • Updated 3 days ago