# KV Caching problem with gemma 3

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkv-caching-problem-with-gemma-3%2F173571%3E%5C
**Category:** 🤗Transformers\
**Created:** [February 17, 2026, 1:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkv-caching-problem-with-gemma-3%2F173571 "2026-02-17T13:38:16Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![zzzz567](https://avatars.discourse-cdn.com/v4/letter/z/bbce88/32.png) [@zzzz567](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzzzz567)%5C
**Post date:** [February 17, 2026, 1:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkv-caching-problem-with-gemma-3%2F173571%2F1 "2026-02-17T13:38:16Z")

</div>

Hi I have fine-tuned a gemma 3 270M model with unsloth.

I am now trying to implement a caching mechanism for my system prompt. Went through HF doc and particularly this [section](https://huggingface.co/docs/transformers/en/kv_cache#prefill-a-cache-prefix-caching).

However, I am getting the following the error when running:

```auto
from transformers.cache_utils import StaticCache
import torch
from pathlib import Path
import json
from unsloth import FastLanguageModel
import time
import copy

model_gemma, tokenizer_gemma = FastLanguageModel.from_pretrained(
    model_name = "gemma_3_lora", # YOUR MODEL YOU USED FOR TRAINING
    max_seq_length = 2048,
    load_in_4bit = False,
)

PROMPT_SYSTEM = """
### Instruction:
Extract metadata from the document text below.
You must output a VALID JSON object. Do not output lists, markdown, or conversational text.

Required Keys:
- "document_number": The drawing ID
- "document_title": The main title
- "document_revision": Revision code (e.g., C01)
- "document_date": YYYY-MM-DD
"""

PROMPT_INPUT = """
### Input:
{context}

### Response:
"""

cache_sys = StaticCache(config=model_gemma.config, max_cache_len=1024, device=model_gemma.device, dtype=model_gemma.dtype)
inputs_sys = tokenizer_gemma(PROMPT_SYSTEM, return_tensors = "pt").to("cuda")

with torch.no_grad():
    cache_sys = model_gemma(
        **inputs_sys,
        past_key_values=cache_sys
    ).past_key_values

text_input_template = tokenizer_gemma.apply_chat_template(
            [
            {"role" : "system", "content": PROMPT_SYSTEM},
            {"role" : "user", "content": PROMPT_INPUT.format(context="This is some fake data")}
            ],
            tokenize = False,
            add_generation_prompt = True).removeprefix('<bos>')

tokened_input_text = tokenizer_gemma(text_input_template, return_tensors = "pt").to("cuda")

past_key_values = copy.deepcopy(cache_sys)

outputs = model_gemma.generate(
    **tokened_input_text,
    temperature = 1, 
    top_p = 0.95, 
    top_k = 64,
    past_key_values=past_key_values
)
input_length = tokened_input_text["input_ids"].shape[1]
generated_tokens = outputs[:, input_length:]
response_variable = tokenizer_gemma.decode(generated_tokens[0], skip_special_tokens=True)

```

I get the following error from the generate function

```auto
ValueError: Passing both \`cache_implementation\` (used to initialize certain caches) and \`past_key_values\` (a Cache object) is unsupported. Please use only one of the two.

```

I have tried to pass cache\_implementation as None but still the same error

I am using `transformers==4.57.6`.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [February 17, 2026, 4:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkv-caching-problem-with-gemma-3%2F173571%2F2 "2026-02-17T16:42:05Z")

</div>

Gemma 3 family models have some quirk…

* * *

## What is causing your `ValueError`

That error is thrown by `transformers` when **both** of these are true at generation time:

1. `generation_config.cache_implementation` is **not `None`** (Transformers will try to initialize/manage a cache itself), **and**
2. you pass `past_key_values` as a **Cache object** (e.g., `StaticCache`), meaning _you_ are managing the cache.

Transformers explicitly rejects that combination and raises exactly your message. ([GitHub](https://raw.githubusercontent.com/huggingface/transformers/v4.57.6/src/transformers/generation/utils.py "raw.githubusercontent.com"))

### Why this happens “by default” with Gemma 3

Many Gemma 3 model repos ship a `generation_config.json` with:

```json
"cache_implementation": "hybrid"

```

So even if you never set `cache_implementation` in code, `model.generation_config.cache_implementation` starts as `"hybrid"`. ([Hugging Face](https://huggingface.co/unsloth/gemma-3-270m-it-qat-unsloth-bnb-4bit/blob/main/generation_config.json "generation\_config.json · unsloth/gemma-3-270m-it-qat-unsloth-bnb-4bit at main"))

That’s why `generate(past_key_values=StaticCache(...))` immediately errors: you’re passing a user cache while the model’s generation config says “use hybrid cache”. ([GitHub](https://raw.githubusercontent.com/huggingface/transformers/v4.57.6/src/transformers/generation/utils.py "raw.githubusercontent.com"))

* * *

## Why “I tried `cache_implementation=None`” often still fails with Unsloth

Unsloth commonly patches `model.generate` with a fast wrapper (`unsloth_base_fast_generate`) that **overwrites caching settings**. In the wrapper, Unsloth computes a cache type (`"static"` / `"hybrid"` / `None`) and then sets:

- `kwargs["generation_config"].cache_implementation = cache_implementation` (if a generation\_config is passed), **or**
- `kwargs["cache_implementation"] = cache_implementation` (otherwise),

then calls `self._old_generate(...)`. ([GitHub](https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/models/vision.py "raw.githubusercontent.com"))

So even if you set `cache_implementation=None`, the wrapper can set it back to `"static"`/`"hybrid"`, and Transformers will again see both `cache_implementation` and `past_key_values` → same `ValueError`. ([GitHub](https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/models/vision.py "raw.githubusercontent.com"))

* * *

## Solutions (pick one)

### Solution 1 (recommended for prefix-caching): disable Unsloth fast generation

Set before importing `unsloth`:

```python
import os
os.environ["UNSLOTH_DISABLE_FAST_GENERATION"] = "1"

```

Unsloth documents this flag. ([unsloth.ai](https://unsloth.ai/docs/basics/unsloth-environment-flags "Unsloth Environment Flags | Unsloth Documentation"))

Then you can safely do:

- `generation_config.cache_implementation = None`
- pass `past_key_values=StaticCache(...)`

### Solution 2: bypass the wrapper and call the original generate

If Unsloth already patched `generate`, it calls `self._old_generate(...)` internally. You can call that directly so Unsloth doesn’t re-inject `cache_implementation`. ([GitHub](https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/models/vision.py "raw.githubusercontent.com"))

Pattern:

```python
gen = model_gemma._old_generate if hasattr(model_gemma, "_old_generate") else model_gemma.generate
gen(**inputs, generation_config=gen_cfg, past_key_values=cache, ...)

```

### Solution 3: stop using a user cache (use only `cache_implementation`)

If you remove `past_key_values=StaticCache(...)` and rely purely on `cache_implementation`, Transformers will manage caching internally. This avoids the error, but it does **not** give you system-prompt _prefix reuse across independent requests_ in the same way as a persisted prefilled cache.

* * *

## Additional issue in your specific code: the cached prefix does not match your chat prompt

You prefill the cache with:

```python
inputs_sys = tokenizer_gemma(PROMPT_SYSTEM, ...)

```

But later you generate from a prompt produced by `apply_chat_template(...)` containing a system message + user message. Those tokens are **not the same prefix** as the raw `PROMPT_SYSTEM` string.

For prefix caching to be correct/beneficial, the cached prefix must match the **exact token sequence** at the start of the later prompt. Hugging Face’s prefix caching example works because the “INITIAL\_PROMPT” is exactly the prefix of the later prompts. ([Hugging Face](https://huggingface.co/docs/transformers/en/kv_cache "Cache strategies"))

**Fix:** prefill using the same chat template with only the system message, then generate from system+user.

* * *

## Another important Gemma 3 constraint: “unprocessed input\_ids” with caches

Gemma 3’s model docs state that when `past_key_values` are used, the user is expected to pass only the **unprocessed** `input_ids` (tokens not already covered by the cache). ([Hugging Face](https://huggingface.co/docs/transformers/en/model_doc/gemma3 "Gemma 3"))

Transformers `generate()` usually slices inputs appropriately, but you can still run into edge cases (especially if your cached prefix and the prompt don’t align, or if the “suffix length” becomes zero).

* * *

## Minimal “fix shape” for your original snippet

Key changes:

- make `generation_config.cache_implementation=None` for the call
- prefill using chat template system-only
- disable Unsloth fast generation _or_ call `_old_generate`

```python
import os
os.environ["UNSLOTH_DISABLE_FAST_GENERATION"] = "1" # must be set before importing unsloth :contentReference[oaicite:9]{index=9}

import copy, torch
from transformers.cache_utils import StaticCache
from unsloth import FastLanguageModel

model, tok = FastLanguageModel.from_pretrained("gemma_3_lora", max_seq_length=2048, load_in_4bit=False)
model.eval()

# 1) Build SYSTEM-only prefix using chat template (token-exact prefix)
sys_inputs = tok.apply_chat_template(
    [{"role":"system","content": PROMPT_SYSTEM}],
    add_generation_prompt=False,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

# 2) Prefill cache with that prefix
cache = StaticCache(config=model.config, max_cache_len=2048, device=model.device, dtype=model.dtype)
with torch.no_grad():
    cache = model(**sys_inputs, past_key_values=cache, use_cache=True).past_key_values

# 3) Build FULL prompt (system + user)
full_inputs = tok.apply_chat_template(
    [{"role":"system","content": PROMPT_SYSTEM},
     {"role":"user","content": PROMPT_INPUT.format(context="This is some fake data")}],
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

# 4) Ensure cache_implementation is None when passing a Cache object :contentReference[oaicite:10]{index=10}
gen_cfg = copy.deepcopy(model.generation_config)
gen_cfg.cache_implementation = None

# 5) Generate (unpatched generate, because fast generation disabled)
out = model.generate(**full_inputs, generation_config=gen_cfg, past_key_values=copy.deepcopy(cache), max_new_tokens=128)

```

* * *

## Summary of “causes → fixes”

- **Cause A:** Gemma 3 repos often set `"cache_implementation": "hybrid"` in `generation_config.json` ([Hugging Face](https://huggingface.co/unsloth/gemma-3-270m-it-qat-unsloth-bnb-4bit/blob/main/generation_config.json "generation\_config.json · unsloth/gemma-3-270m-it-qat-unsloth-bnb-4bit at main"))  
**Fix:** pass a `GenerationConfig` where `cache_implementation=None` when using `past_key_values` ([GitHub](https://raw.githubusercontent.com/huggingface/transformers/v4.57.6/src/transformers/generation/utils.py "raw.githubusercontent.com"))

- **Cause B:** Unsloth fast generation wrapper can forcibly set `cache_implementation` (static/hybrid) before calling the original generate ([GitHub](https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/models/vision.py "raw.githubusercontent.com"))  
**Fix:** `UNSLOTH_DISABLE_FAST_GENERATION=1` ([unsloth.ai](https://unsloth.ai/docs/basics/unsloth-environment-flags "Unsloth Environment Flags | Unsloth Documentation")) or call `_old_generate` ([GitHub](https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/models/vision.py "raw.githubusercontent.com"))

- **Cause C (logic/prefix mismatch):** you cached raw `PROMPT_SYSTEM` tokens but generated from chat-template tokens  
**Fix:** prefill cache using system-only `apply_chat_template(...)` so the cached tokens exactly match the start of later prompts ([Hugging Face](https://huggingface.co/docs/transformers/en/kv_cache "Cache strategies"))

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [February 18, 2026, 10:22am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fkv-caching-problem-with-gemma-3%2F173571%2F3 "2026-02-18T10:22:06Z")

</div>

Closed on February 18, 2026, 10:22am UTC

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.