---
license: apache-2.0
base_model:
  - XHToken/Spark-X2.6-4B
base_model_relation: quantized
pipeline_tag: text-generation
library_name: gguf
language:
  - en
  - zh
tags:
  - gguf
  - safetensors
  - fp8
  - nvfp4
  - spark
  - spark2_5
  - spark-x2.6
  - text-generation
  - reasoning
  - agent
  - llama-cpp
  - ollama
---

[![🤗 Model Requests](https://img.shields.io/badge/%F0%9F%A4%97_Model_Requests-Submit_Here-f59e0b?style=for-the-badge&labelColor=1f2937)](https://abenzerps.com/request-a-model/)

# Spark-X2.6-4B Quantized

GGUF and Safetensors (FP8, NVFP4) quantized variants of [XHToken/Spark-X2.6-4B](https://huggingface.co/XHToken/Spark-X2.6-4B) for local execution, agent workflows, and reasoning.

**Spark-X2.6-4B** by [XHToken](https://huggingface.co/XHToken) is a lightweight 4B language model optimized for on-device agents, coding, and tool use. Featuring a hybrid-attention design with up to a 1M context window and controllable thinking budgets (`low` / `high`), it matches or outperforms much larger 9B–12B models on standard agent and coding benchmarks while remaining exceptionally fast.

## Benchmark

![Spark-X2.6-4B Benchmark](assets/benchmark.png)

## Quantized Files

| **Quantization** | **File** | **Size** |
| --- | --- | ---: |
| FP8 | [**spark-x2.6-4b-fp8.safetensors**](https://huggingface.co/abenzerps/Spark-X2.6-4B-Quantized/blob/main/spark-x2.6-4b-fp8.safetensors) | 4.14 GB |
| NVFP4 | [**spark-x2.6-4b-NVFP4.safetensors**](https://huggingface.co/abenzerps/Spark-X2.6-4B-Quantized/blob/main/spark-x2.6-4b-NVFP4.safetensors) | 2.60 GB |
| BF16 | [**Spark-X2.6-4B-BF16.gguf**](https://huggingface.co/abenzerps/Spark-X2.6-4B-Quantized/blob/main/Spark-X2.6-4B-BF16.gguf) | 7.66 GB |
| Q8_0 | [**Spark-X2.6-4B-Q8_0.gguf**](https://huggingface.co/abenzerps/Spark-X2.6-4B-Quantized/blob/main/Spark-X2.6-4B-Q8_0.gguf) | 4.10 GB |
| Q6_K | [**Spark-X2.6-4B-Q6_K.gguf**](https://huggingface.co/abenzerps/Spark-X2.6-4B-Quantized/blob/main/Spark-X2.6-4B-Q6_K.gguf) | 3.20 GB |
| Q5_K_M | [**Spark-X2.6-4B-Q5_K_M.gguf**](https://huggingface.co/abenzerps/Spark-X2.6-4B-Quantized/blob/main/Spark-X2.6-4B-Q5_K_M.gguf) | 2.80 GB |
| Q4_K_M | [**Spark-X2.6-4B-Q4_K_M.gguf**](https://huggingface.co/abenzerps/Spark-X2.6-4B-Quantized/blob/main/Spark-X2.6-4B-Q4_K_M.gguf) | 2.47 GB |
| Q4_0 | [**Spark-X2.6-4B-Q4_0.gguf**](https://huggingface.co/abenzerps/Spark-X2.6-4B-Quantized/blob/main/Spark-X2.6-4B-Q4_0.gguf) | 2.24 GB |

**Q4_K_M** is recommended for the best balance of size and quality for GGUF. **NVFP4** and **FP8** offer high performance on supported NVIDIA GPUs.

## Usage

### 1. llama.cpp / llama-server (GGUF)

#### Interactive Chat via CLI
Run multi-turn interactive conversation directly in your terminal:

```bash
./llama-cli -m Spark-X2.6-4B-Q4_K_M.gguf \
    -cnv \
    -c 8192 \
    -n 512 \
    --temp 0.6 \
    -t 8
```

#### OpenAI-Compatible API Server
Start `llama-server` to serve OpenAI-compatible `/v1/chat/completions` endpoints:

```bash
./llama-server -m Spark-X2.6-4B-Q4_K_M.gguf \
    -c 8192 \
    --port 8080
```

Query via curl:

```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "Hello! What can you help me with?"
      }
    ],
    "temperature": 0.6
  }'
```

### 2. Ollama (GGUF)

Create a `Modelfile`:

```dockerfile
FROM ./Spark-X2.6-4B-Q4_K_M.gguf

TEMPLATE """{{- if .System }}<｜start▁of▁sentence｜><|System|>{{ .System }}
{{ end }}{{- range .Messages }}<｜start▁of▁sentence｜><|{{ if eq .Role "user" }}User{{ else }}Bot{{ end }}|>{{ .Content }}<｜end▁of▁sentence｜>{{- end }}{{- if .Prompt }}<｜start▁of▁sentence｜><|User|>{{ .Prompt }}<｜end▁of▁sentence｜><｜start▁of▁sentence｜><|Bot|><think>
{{ end }}"""

PARAMETER stop "<｜end▁of▁sentence｜>"
PARAMETER stop "<｜start▁of▁sentence｜>"
PARAMETER temperature 0.6
```

Build and run:

```bash
ollama create spark-x2.6 -f Modelfile
ollama run spark-x2.6
```

### 3. NVIDIA FP8 & NVFP4 (Safetensors)

Load the quantized weights directly via `safetensors` for TensorCore-accelerated pipelines (such as `comfy-kitchen`, `vLLM`, or custom inference):

```python
import safetensors.torch
import torch

# Load FP8 (torch.float8_e4m3fn with per-tensor scales)
fp8_weights = safetensors.torch.load_file("spark-x2.6-4b-fp8.safetensors")

# Load NVFP4 (packed 4-bit with block-wise scales)
nvfp4_weights = safetensors.torch.load_file("spark-x2.6-4b-NVFP4.safetensors")
```

## Source and Build

- **Source model:** [XHToken/Spark-X2.6-4B](https://huggingface.co/XHToken/Spark-X2.6-4B)
- **Architecture:** `Spark2_5ForCausalLM` (`spark2_5` GGUF architecture)
- **Conversion & Quantization:** Upstream [llama.cpp](https://github.com/ggml-org/llama.cpp) & [comfy-kitchen](https://github.com/comfyanonymous/comfy-kitchen)
- **License:** Apache 2.0
- **Checksums:** [SHA256SUMS](./SHA256SUMS)