# Cache size much larger than \`dh cache scan\` shows

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcache-size-much-larger-than-dh-cache-scan-shows%2F169348%3E%5C
**Category:** 🤗Hub\
**Created:** [October 23, 2025, 12:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcache-size-much-larger-than-dh-cache-scan-shows%2F169348 "2025-10-23T00:19:56Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [October 23, 2025, 4:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcache-size-much-larger-than-dh-cache-scan-shows%2F169348%2F2 "2025-10-23T04:46:15Z")

</div>

oh…

* * *

Diagnosis: your numbers make sense. `hf cache scan` only reports the **Hub** cache; `ncdu` sums **Hub**  **plus** the Datasets Arrow/processed cache **plus** temporary download caches. 55 GB in `hub/` + ~153 GB in `datasets/` ≈ your 217 GB. This is normal for large audio datasets with transforms and repeated variant loads. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/en/guides/manage-cache "Understand caching"))

# a) Corrupt cache vs “expected extra space”

Mostly expected extra space, not corruption.

- Two caches exist:

- Interrupted runs may leave partials, but the usual bloat is from Arrow shards and extracted archives created during prepare steps. You can clean surgically:

- If you changed the language list, `datasets` treats it as a different configuration/fingerprint. That creates **new** processed caches even when raw files are reused. ([Hugging Face](https://huggingface.co/docs/datasets/v1.4.0/processing.html "Processing data in a Dataset"))

# b) Why `~/.cache/hub` is big though you “only used datasets”

Because datasets are also stored under the **Hub** cache. The Hub cache is shared by all HF libraries and has top-level `DATASETS` beside `MODELS`. `load_dataset(...)` fetches scripts and often the raw data from the Hub, which grows `~/.cache/huggingface/hub`. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/en/guides/manage-cache "Understand caching"))

# c) Why the bar sits at 100% while disk and network still grow

The tqdm bar usually tracks the **download** phase only. After 100% it may still be:

- Extracting archives and verifying content.
- Converting to Arrow and writing shards to `~/.cache/huggingface/datasets/...`.
- Building indices and split files.  
These steps can be long for audio and can look like a “stuck at 100%” bar. Users often report “100% but still running” during filtering/mapping or when using multiple processes. ([Hugging Face](https://huggingface.co/docs/datasets/en/cache "Cache management"))

* * *

## What to do now (surgical, keeps good files)

1. Inspect Hub usage only

```bash
# Hub-only report; excludes ~/.cache/huggingface/datasets
hf cache scan
hf cache scan --dir ~/.cache/huggingface/hub

```

([Hugging Face](https://huggingface.co/docs/huggingface_hub/v0.33.2/en/guides/manage-cache "Understand caching"))

1. Prune **only** MLCommons/ml\_spoken\_words from the Hub cache

```bash
# remove this dataset’s Hub entries only
hf cache delete dataset/MLCommons/ml_spoken_words -y
# or, older CLI
huggingface-cli scan-cache --delete --repos dataset/MLCommons/ml_spoken_words

```

([Hugging Face](https://huggingface.co/docs/huggingface_hub/v0.33.2/en/guides/manage-cache "Understand caching"))

1. Prune processed Arrow for **this** dataset

```python
from datasets import load_dataset
ds = load_dataset("MLCommons/ml_spoken_words", languages=["ar","tt"])
print(ds.cleanup_cache_files()) # returns count of removed files

```

([Hugging Face](https://huggingface.co/docs/datasets/v1.12.0/cache.html "Cache management — datasets 1.12.0 documentation"))

1. Pin caches to the large partition so future cleanup is trivial

```bash
# Hub cache root (affects hub/*, incl. chunk-cache/shard-cache)
export HF_HOME="/data/hf"

# Datasets Arrow cache
export HF_DATASETS_CACHE="/data/hf/datasets"

```

([Hugging Face](https://huggingface.co/docs/huggingface_hub/v0.33.2/en/guides/manage-cache "Understand caching"))

1. If a run was interrupted and you want a clean fetch for **just** this dataset

```python
from datasets import load_dataset, GenerateMode
ds = load_dataset("MLCommons/ml_spoken_words",
                  languages=["ar","tt"],
                  download_mode=GenerateMode.FORCE_REDOWNLOAD)

```

([Hugging Face](https://huggingface.co/docs/datasets/v1.12.0/cache.html "Cache management — datasets 1.12.0 documentation"))

1. If disk is tight, stream instead of caching to disk

```python
from datasets import load_dataset
ds = load_dataset("MLCommons/ml_spoken_words",
                  languages=["ar","tt"],
                  streaming=True, split="train")
first = next(iter(ds))

```

This avoids Arrow writes and large `datasets/` growth. ([Hugging Face](https://huggingface.co/docs/datasets/en/stream "Stream"))

1. Keep the language list stable  
Changing `languages=[...]` changes the config name (e.g., `"ar+tt"`), which triggers a new processed cache. Decide the set once before a full prepare. ([GitHub](https://github.com/huggingface/datasets/pull/3666.patch "https://github.com/huggingface/datasets/pull/3666...."))

* * *

## Why your 60.2 GB vs 217 GB mismatch is expected

- `hf cache scan` summarizes **Hub** only. It excludes `~/.cache/huggingface/datasets` where Arrow and extracted files live. ([GitHub](https://github.com/huggingface/huggingface_hub/issues/2218 "huggingface-cli scan-cache doesn't capture cached datasets"))
- The datasets cache stores Arrow shards and `downloads/` and `downloads/extracted/`, which can exceed the raw download size. ([Hugging Face](https://huggingface.co/docs/datasets/en/cache "Cache management"))
- During active transfers you also have `chunk-cache/` and `shard-cache/` inside the Hub cache. These raise the live total and are safe to remove when done. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/v0.33.4/guides/manage-cache "Understand caching"))

* * *

## Short curated references

**Official docs**

- Hub cache layout, CLI, and transient `chunk-cache`/`shard-cache`. Explains Hub vs Datasets and safe deletions. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/v0.33.4/guides/manage-cache "Understand caching"))
- Datasets cache guide. Explains Arrow shards, `downloads/`, `extracted/`, cleanup, and moving the cache. ([Hugging Face](https://huggingface.co/docs/datasets/en/cache "Cache management"))
- Streaming mode. Use datasets without local disk growth. ([Hugging Face](https://huggingface.co/docs/datasets/en/stream "Stream"))

**Issues and forum threads**

- `scan-cache` doesn’t include the Datasets cache. Confirms the undercount. ([GitHub](https://github.com/huggingface/huggingface_hub/issues/2218 "huggingface-cli scan-cache doesn't capture cached datasets"))
- Progress shows 100% but work continues during map/filter or multithreaded steps. ([Hugging Face Forums](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-filter-map-hangs-when-multithreading%2F36967 "Datasets filter/map hangs when multithreading"))

**Dataset-specific**

- MSWC dataset card and loader usage with multiple languages. ([Hugging Face](https://huggingface.co/datasets/MLCommons/ml_spoken_words "MLCommons/ml\_spoken\_words · Datasets at Hugging Face"))
- Implementation shows the config name is the `"+".join(languages)`, so different language sets create different processed caches. ([GitHub](https://github.com/huggingface/datasets/pull/3666.patch "https://github.com/huggingface/datasets/pull/3666...."))

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcache-size-much-larger-than-dh-cache-scan-shows%2F169348)._