# \`load\_dataset\` results in OOM

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-dataset-results-in-oom%2F94028%3E%5C
**Category:** 🤗Datasets\
**Created:** [June 25, 2024, 4:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-dataset-results-in-oom%2F94028 "2024-06-25T16:08:49Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![sharvil-lmnt](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sharvil-lmnt/32/28616_2.png) [@sharvil-lmnt](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsharvil-lmnt)%5C
**Post date:** [June 25, 2024, 4:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-dataset-results-in-oom%2F94028%2F1 "2024-06-25T16:08:49Z")

</div>

I have a few large (private) datasets that I want to concatenate. I can’t seem to load them without OOMing. Here’s an example of what I’m doing:

```auto
import datasets

all_datasets = []
for name in ['meow/private1', 'meow/private2', 'meow/private3', 'meow/private4']:
  ds = datasets.load_dataset(name)
  all_datasets.append(ds)
all_datasets = datasets.concatenate_datasets(all_datasets) # This code is not reached

```

The datasets do not have a custom loading script. They’re cached locally as a set of Arrow files. From what I can tell in the docs, the files should be mmaped into my process and demand-loaded from disk. Notice that I’m not even touching the data so I’m not sure what’s being loaded into memory. I’ve also tried setting `datasets.config.IN_MEMORY_MAX_SIZE` to 1GB with no luck.

Is my understanding incorrect? Does `load_dataset` always attempt to read the entire dataset into memory? Is there any way to load these datasets as map-style datasets (i.e. not `IterableDataset`)?