# Cache size much larger than \`dh cache scan\` shows
**URL:**
**Author:**  [@Jacob-Stevens-Haas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJacob-Stevens-Haas)%5C
**Post date:** [October 23, 2025, 12:19am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcache-size-much-larger-than-dh-cache-scan-shows%2F169348%2F1 "2025-10-23T00:19:56Z")
Hi, I was trying to download three languages of MLCommons/ml\_spoken\_words. When I ran out of space, I deleted some things and tried with just two of the languages. I deleted more things, moved my OS to a new, much larger partition, and continued to try to download the datasets. When I run `hf cache scan`, It shows:
```auto
MLCommons/ml_spoken_words dataset 60.2G
```
On the other hand, when I run `ncdu`, it shows 217 GB for ~/.cache/huggingface, 55 GB in hub/ and 153 GB in datasets/
Meanwhile, `load_dataset` has a progress bar that has shown 100% for a while, but the cache keeps growing and `nethogs` shows that it’s using as much bandwidth as it can. I know I can manually delete the cache and try again, but
a) Do I have a large amount of corrupt cache from the interrupted downloads (and how do I clear it without the good downloads), or does huggingface use extra disk space (e.g. to block out a space for transforms)?
b) Why do I have so much in ~/.cache/hub when I’ve only ever downloaded a dataset, not a model?
c) what is `load_dataset` doing that takes so long after it reaches 100%? Is this a progress bar issue, or is this potentially a memory issue caused by interrupted downloads?
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [October 23, 2025, 4:46am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcache-size-much-larger-than-dh-cache-scan-shows%2F169348%2F2 "2025-10-23T04:46:15Z")
oh…
* * *
Diagnosis: your numbers make sense. `hf cache scan` only reports the **Hub** cache; `ncdu` sums **Hub** **plus** the Datasets Arrow/processed cache **plus** temporary download caches. 55 GB in `hub/` + ~153 GB in `datasets/` ≈ your 217 GB. This is normal for large audio datasets with transforms and repeated variant loads. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/en/guides/manage-cache "Understand caching"))
# a) Corrupt cache vs “expected extra space”
Mostly expected extra space, not corruption.
- Two caches exist:
- Interrupted runs may leave partials, but the usual bloat is from Arrow shards and extracted archives created during prepare steps. You can clean surgically:
- If you changed the language list, `datasets` treats it as a different configuration/fingerprint. That creates **new** processed caches even when raw files are reused. ([Hugging Face](https://huggingface.co/docs/datasets/v1.4.0/processing.html "Processing data in a Dataset"))
# b) Why `~/.cache/hub` is big though you “only used datasets”
Because datasets are also stored under the **Hub** cache. The Hub cache is shared by all HF libraries and has top-level `DATASETS` beside `MODELS`. `load_dataset(...)` fetches scripts and often the raw data from the Hub, which grows `~/.cache/huggingface/hub`. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/en/guides/manage-cache "Understand caching"))
# c) Why the bar sits at 100% while disk and network still grow
The tqdm bar usually tracks the **download** phase only. After 100% it may still be:
- Extracting archives and verifying content.
- Converting to Arrow and writing shards to `~/.cache/huggingface/datasets/...`.
- Building indices and split files.
These steps can be long for audio and can look like a “stuck at 100%” bar. Users often report “100% but still running” during filtering/mapping or when using multiple processes. ([Hugging Face](https://huggingface.co/docs/datasets/en/cache "Cache management"))
* * *
## What to do now (surgical, keeps good files)
1. Inspect Hub usage only
```bash
# Hub-only report; excludes ~/.cache/huggingface/datasets
hf cache scan
hf cache scan --dir ~/.cache/huggingface/hub
```
([Hugging Face](https://huggingface.co/docs/huggingface_hub/v0.33.2/en/guides/manage-cache "Understand caching"))
1. Prune **only** MLCommons/ml\_spoken\_words from the Hub cache
```bash
# remove this dataset’s Hub entries only
hf cache delete dataset/MLCommons/ml_spoken_words -y
# or, older CLI
huggingface-cli scan-cache --delete --repos dataset/MLCommons/ml_spoken_words
```
([Hugging Face](https://huggingface.co/docs/huggingface_hub/v0.33.2/en/guides/manage-cache "Understand caching"))
1. Prune processed Arrow for **this** dataset
```python
from datasets import load_dataset
ds = load_dataset("MLCommons/ml_spoken_words", languages=["ar","tt"])
print(ds.cleanup_cache_files()) # returns count of removed files
```
([Hugging Face](https://huggingface.co/docs/datasets/v1.12.0/cache.html "Cache management — datasets 1.12.0 documentation"))
1. Pin caches to the large partition so future cleanup is trivial
```bash
# Hub cache root (affects hub/*, incl. chunk-cache/shard-cache)
export HF_HOME="/data/hf"
# Datasets Arrow cache
export HF_DATASETS_CACHE="/data/hf/datasets"
```
([Hugging Face](https://huggingface.co/docs/huggingface_hub/v0.33.2/en/guides/manage-cache "Understand caching"))
1. If a run was interrupted and you want a clean fetch for **just** this dataset
```python
from datasets import load_dataset, GenerateMode
ds = load_dataset("MLCommons/ml_spoken_words",
languages=["ar","tt"],
download_mode=GenerateMode.FORCE_REDOWNLOAD)
```
([Hugging Face](https://huggingface.co/docs/datasets/v1.12.0/cache.html "Cache management — datasets 1.12.0 documentation"))
1. If disk is tight, stream instead of caching to disk
```python
from datasets import load_dataset
ds = load_dataset("MLCommons/ml_spoken_words",
languages=["ar","tt"],
streaming=True, split="train")
first = next(iter(ds))
```
This avoids Arrow writes and large `datasets/` growth. ([Hugging Face](https://huggingface.co/docs/datasets/en/stream "Stream"))
1. Keep the language list stable
Changing `languages=[...]` changes the config name (e.g., `"ar+tt"`), which triggers a new processed cache. Decide the set once before a full prepare. ([GitHub](https://github.com/huggingface/datasets/pull/3666.patch "https://github.com/huggingface/datasets/pull/3666...."))
* * *
## Why your 60.2 GB vs 217 GB mismatch is expected
- `hf cache scan` summarizes **Hub** only. It excludes `~/.cache/huggingface/datasets` where Arrow and extracted files live. ([GitHub](https://github.com/huggingface/huggingface_hub/issues/2218 "huggingface-cli scan-cache doesn't capture cached datasets"))
- The datasets cache stores Arrow shards and `downloads/` and `downloads/extracted/`, which can exceed the raw download size. ([Hugging Face](https://huggingface.co/docs/datasets/en/cache "Cache management"))
- During active transfers you also have `chunk-cache/` and `shard-cache/` inside the Hub cache. These raise the live total and are safe to remove when done. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/v0.33.4/guides/manage-cache "Understand caching"))
* * *
## Short curated references
**Official docs**
- Hub cache layout, CLI, and transient `chunk-cache`/`shard-cache`. Explains Hub vs Datasets and safe deletions. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/v0.33.4/guides/manage-cache "Understand caching"))
- Datasets cache guide. Explains Arrow shards, `downloads/`, `extracted/`, cleanup, and moving the cache. ([Hugging Face](https://huggingface.co/docs/datasets/en/cache "Cache management"))
- Streaming mode. Use datasets without local disk growth. ([Hugging Face](https://huggingface.co/docs/datasets/en/stream "Stream"))
**Issues and forum threads**
- `scan-cache` doesn’t include the Datasets cache. Confirms the undercount. ([GitHub](https://github.com/huggingface/huggingface_hub/issues/2218 "huggingface-cli scan-cache doesn't capture cached datasets"))
- Progress shows 100% but work continues during map/filter or multithreaded steps. ([Hugging Face Forums](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-filter-map-hangs-when-multithreading%2F36967 "Datasets filter/map hangs when multithreading"))
**Dataset-specific**
- MSWC dataset card and loader usage with multiple languages. ([Hugging Face](https://huggingface.co/datasets/MLCommons/ml_spoken_words "MLCommons/ml\_spoken\_words · Datasets at Hugging Face"))
- Implementation shows the config name is the `"+".join(languages)`, so different language sets create different processed caches. ([GitHub](https://github.com/huggingface/datasets/pull/3666.patch "https://github.com/huggingface/datasets/pull/3666...."))
---
**Author:**  [@Jacob-Stevens-Haas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJacob-Stevens-Haas)%5C
**Post date:** [October 23, 2025, 5:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcache-size-much-larger-than-dh-cache-scan-shows%2F169348%2F3 "2025-10-23T17:58:16Z")
Wait but `load_dataset` takes a long time. Are you telling me I need to load a dataset in order to delete it’s cache?
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [October 24, 2025, 1:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcache-size-much-larger-than-dh-cache-scan-shows%2F169348%2F4 "2025-10-24T01:23:46Z")
To delete the cache for a specific model or dataset, you can also locate and delete it directly using your OS’s file manager. However, it can be hard to find…😅
Personally, I recommend using the HF CLI as it’s the most reliable method.
* * *
No. You do not need to reload the dataset to delete its cache.
How to delete without loading:
1. Hub cache (the 60 GB “scan-cache” sees)
Delete the repo from the Hub cache directly. No Python, no dataset load.
```bash
# preview
hf cache ls --filter "repo_id==dataset/MLCommons/ml_spoken_words"
hf cache rm dataset/MLCommons/ml_spoken_words --dry-run
# delete
hf cache rm dataset/MLCommons/ml_spoken_words -y
# if your cache lives elsewhere
hf cache rm dataset/MLCommons/ml_spoken_words -y --cache-dir /path/to/hf/hub
```
This is the supported way to surgically remove a dataset repo from the Hub cache. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/en/guides/manage-cache "Understand caching"))
1. Datasets processed cache (the ~153 GB under `~/.cache/huggingface/datasets`)
You can remove those Arrow/processed files by path. No need to construct a `Dataset` in Python.
```bash
# find the directories for this dataset
find ~/.cache/huggingface/datasets -maxdepth 3 -type d -iname '*ml_spoken_words*' -print
# common space hogs you can delete safely
rm -rf ~/.cache/huggingface/datasets/downloads # raw archives
rm -rf ~/.cache/huggingface/datasets/downloads/extracted # extracted archives
# remove only this dataset's processed shards (after confirming paths via `find`)
rm -rf ~/.cache/huggingface/datasets/*ml_spoken_words*
```
Hugging Face’s Datasets docs and forum confirm: processed caches live under `~/.cache/huggingface/datasets`, and it is safe to delete `downloads/` and dataset-specific folders when you want to reclaim space. ([Hugging Face](https://huggingface.co/docs/datasets/cache "Cache management"))
Notes and alternatives:
- The `cleanup_cache_files()` API does require a `Dataset` object, which implies loading, so skip it if the load is slow and just delete by path as above. The method exists, but it is optional. ([Hugging Face](https://huggingface.co/docs/datasets/v1.12.0/package_reference/main_classes.html "Main classes — datasets 1.12.0 documentation"))
- If you prefer a lighter Python route, you can still avoid a full prepare by cleaning the Hub cache with the CLI, then delete the Datasets cache directories by path; this matches the official cache model split: Hub cache at `~/.cache/huggingface/hub`, Datasets cache at `~/.cache/huggingface/datasets`. ([Hugging Face](https://huggingface.co/docs/datasets/v3.0.0/en/cache "Cache management"))
Summary: use `hf cache rm ...` for the Hub cache, and delete the dataset’s folders under `~/.cache/huggingface/datasets` for processed data. No dataset reload required. ([Hugging Face](https://huggingface.co/docs/huggingface_hub/en/guides/manage-cache "Understand caching"))