# Storing and loading KV cache
**URL:**
**Author:**  [@oran-sh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Foran-sh)%5C
**Post date:** [July 23, 2024, 9:19pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstoring-and-loading-kv-cache%2F99033%2F1 "2024-07-23T21:19:15Z")
Hello,
is there an explicit way to store and later on load KV cache in the models?
Thanks!
---
**Author:**  [@RaushanTurganbay](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRaushanTurganbay)%5C
**Post date:** [July 25, 2024, 4:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstoring-and-loading-kv-cache%2F99033%2F2 "2024-07-25T04:46:26Z")
Hey!
You can reuse a cache object in the next generation steps as follows:
```python
out = model.generate(input_ids, use_cache=True, return_dict_in_generate=True)
past_key_values = out.past_key_values
generated_ids = out.sequences
# Now we can continue generation using cache and already generated tokens
out_continued = model.generate(generated_ids, past_key_values=past_key_values, return_dict_in_generate=True)
continued_generated_ids = out_continued.sequences
```
If you want to save the cache and load it back, we don’t have an explicit way for that. But you can try to save it by saving keys and values, where each of them is a tuple of tensors
```python
keys, values = past_key_values.key_cache, past_key_values.value_cache
torch.save(keys, "keys.pt")
torch.save(values, "values.pt")
# Later you can load it as follows assuming you used the default DynamicCache
from transformers import DynamicCache
past_key_values = DynamicCache()
past_key_values.key_cache = torch.load("keys.pt")
past_key_values.value_cache =values = torch.load("values.pt")
```
Btw, can you share the use case when saving and loading cache in needed. We are now trying to make a unified API for all cache objects, and it will help us to understand common use cases
---
**Author:**  [@oran-sh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Foran-sh)%5C
**Post date:** [August 1, 2024, 7:05am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstoring-and-loading-kv-cache%2F99033%2F3 "2024-08-01T07:05:45Z")
Thanks for the answer @RaushanTurganbay !
I will start with that and try it out.
Regarding use case -
The use case is that we have a lot of long context examples that we repeatedly query, and so it doesn’t make sense to recalculate everything for every query. It makes more sense to reload the KV values for every query.
Does that make sense?
---
**Author:**  [@RaushanTurganbay](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRaushanTurganbay)%5C
**Post date:** [August 1, 2024, 8:03am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstoring-and-loading-kv-cache%2F99033%2F4 "2024-08-01T08:03:41Z")
I see, thanks for explaining. We are making a cache that will inherit from torch Module [here](https://github.com/huggingface/transformers/pull/32159), so after the PR is merged you should be able to copy cache or save with `torch.save`
---
**Author:**  [@oran-sh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Foran-sh)%5C
**Post date:** [August 6, 2024, 1:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstoring-and-loading-kv-cache%2F99033%2F5 "2024-08-06T13:08:37Z")
That’s great, any estimate when this PR will be merged?
---
**Author:**  [@allenwang37](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fallenwang37)%5C
**Post date:** [October 21, 2024, 4:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstoring-and-loading-kv-cache%2F99033%2F6 "2024-10-21T04:35:52Z")
> [@RaushanTurganbay](#):
>
> save with `torch.sa`
@RaushanTurganbay hi, so what is the best practice of kv cache reuse?
---
**Author:**  [@RaushanTurganbay](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FRaushanTurganbay)%5C
**Post date:** [October 21, 2024, 5:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstoring-and-loading-kv-cache%2F99033%2F7 "2024-10-21T17:35:35Z")
For anyone who stumbles upon this thread, here are the docs for reusing cache. But note that it works for most common cache types and will fail if you want to use Offloaded or Quantized Cache
> **[Best Practices for Generation with Cache](https://huggingface.co/docs/transformers/v4.45.2/kv_cache#re-use-cache-to-continue-generation)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.