# Caching only one feature, from a read-only dataset
**URL:**
**Author:**  [@g8a9](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fg8a9)%5C
**Post date:** [March 31, 2025, 7:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcaching-only-one-feature-from-a-read-only-dataset%2F148262%2F1 "2025-03-31T19:04:32Z")
Hey,
I want to add a feature to a large audio dataset before my training starts. In particular, it’s the length in seconds such that my HF trainer can “group\_by\_length” my inputs.
My datasets are all saved locally in a read-only folder (they were saved through `save_to_disk()`).
What’s happening now is that:
- when I load the dataset with `load_from_disk()` that folder is by default used as cache, so any map/filter function fails since I don’t have write access to it (e.g., [this issue](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-from-disk-and-read-only-filesystem%2F54312%2F1))
- If I pass a `cache_filename` with a path where I have write access, the cache files I’m creating are too big, since the whole dataset is cached there (I don’t have enough disk space for that)
- If I remove all the original columns through `remove_columns=` and specify a write-access path, the cache file contains correctly only the feature I’m generating (`length` in this case). However, when I add it back to the dataset through `add_column`, the method internally calls `flatten_indices()`, which again requires writing access to the dataset dir and crashes my script.
Any ideas?
Other constraints that I have are:
- I cannot keep the dataset in memory
- I cannot compute the lengths on the go since I need them for the length grouping sampler
- I cannot afford to compute each sample length every time I run the script since its it takes too long
- I would like to stay within the `datasets` framework since my codebase uses it in several places
---
**Author:**  [@g8a9](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fg8a9)%5C
**Post date:** [April 1, 2025, 11:29am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcaching-only-one-feature-from-a-read-only-dataset%2F148262%2F3 "2025-04-01T11:29:52Z")
I’m sorry, is this response AI-generated?
If possibile, I would try to keep the conversation between humans (and the proposed approach does not address any of my issues 🙂 )
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [April 1, 2025, 4:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcaching-only-one-feature-from-a-read-only-dataset%2F148262%2F4 "2025-04-01T16:38:49Z")
Hi ! maybe you can only keep the lengths in memory, and then concatenate back to the memory mapped (i.e. loaded from disk) dataset containing the audio ?
```python
lengths_ds = ds.map(
compute_length,
remove_columns=ds.column_names,
keep_in_memory=True
)
ds = concatenate_datasets([ds, lengths_ds], axis=1)
```
---
**Author:**  [@g8a9](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fg8a9)%5C
**Post date:** [April 1, 2025, 5:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcaching-only-one-feature-from-a-read-only-dataset%2F148262%2F5 "2025-04-01T17:04:37Z")
Thanks! So, I guess the `concatenate_datasets` does not use any caching, right?
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [April 7, 2025, 10:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcaching-only-one-feature-from-a-read-only-dataset%2F148262%2F6 "2025-04-07T10:26:58Z")
yes correct !
---
**Author:**  [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [April 7, 2025, 10:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcaching-only-one-feature-from-a-read-only-dataset%2F148262%2F7 "2025-04-07T22:27:38Z")
Closed on April 7, 2025, 10:27pm UTC
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.