# How to load parquet to datasets without caching?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-parquet-to-datasets-without-caching%2F19564%3E%5C
**Category:** 🤗Datasets\
**Created:** [June 24, 2022, 7:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-parquet-to-datasets-without-caching%2F19564 "2022-06-24T07:35:25Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![szymonbrych](https://avatars.discourse-cdn.com/v4/letter/s/f6c823/32.png) [@szymonbrych](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fszymonbrych)%5C
**Post date:** [June 24, 2022, 7:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-parquet-to-datasets-without-caching%2F19564%2F1 "2022-06-24T07:35:25Z")

</div>

Hi,  
I have one-off actions when I load parquet to do something.  
I **don’t want to cache anything** , it won’t be reused and I would have lots of garbage to clean.  
I set `datasets.set_caching_enabled(False)` but it is ineffective.  
There doesn’t seem to be an option on the `load_from_parquet` to control this.  
How would one accomplish this?

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [June 24, 2022, 3:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-load-parquet-to-datasets-without-caching%2F19564%2F2 "2022-06-24T15:22:00Z")

</div>

Hi! `datasets.set_caching_enabled(False)` only affects the arrow files created via `.map`, not `load_dataset`. Also, parquet files cannot be zero-copied/memory-mapped efficiently (see [Reading and Writing the Apache Parquet Format — Apache Arrow v8.0.0](https://arrow.apache.org/docs/python/parquet.html#reading-parquet-and-memory-mapping)), so the arrow conversion is the only option for big datasets. Still, if you have enough RAM and want to skip this step to avoid generating a cache file, you can create an in-memory dataset directly from parquet as follows:

```python
from datasets import dataset
import pyarrow.parquet as pq
dset = Dataset(pq.read_table("path/to/parquet/file", memory_map=True))

```