# Why is simply accessing dataset features so slow?
**URL:**
**Author:**  [@mgreenbe](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmgreenbe)%5C
**Post date:** [November 18, 2021, 7:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-simply-accessing-dataset-features-so-slow%2F12041%2F1 "2021-11-18T19:57:18Z")
> See [this Colab notebook](https://colab.research.google.com/drive/11HCIEtF-vgl47uwQw5-xEJyxvdTivkqC?usp=sharing) for runnable versions of the code snippets pasted below.
I’ve been playing with the SQuAD dataset and I’ve noticed that simply accessing features of mapped versions of the dataset is extremely slow:
```python
# Let's play with the validation set.
squad = load_dataset("squad")
ds = squad["validation"]
# standard tokeinzer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
# tokenizer the dataset with standard hyperparameters.
f = lambda x: tokenizer(x["question"], x["context"], max_length=384, stride=128, truncation="only_second", padding="max_length", return_overflowing_tokens=True, return_offsets_mapping=True)
mapped_ds = ds.map(f, batched=True, remove_columns=val_ds.column_names)
```
Now, simply executing the statements – just accessing the features –
```auto
attention_mask = mapped_ds["attention_mask"]
input_ids = mapped_ds["input_ids"]
offset_mapping = mapped_ds["offset_mapping"]
overflow_to_sample_mapping = mapped_ds["overflow_to_sample_mapping"]
token_type_ids = mapped_ds["token_type_ids"]
```
takes about **12 seconds**. Why does this take so long? My understanding from the docs is that an HF dataset is stored as a collection of feature columns. (Arrow table format?) Were this so, each assignment in the above snippet should just be assigning a pointer. There’s no data copying happening here, right?
Digging a little deeper, assigning `attention_mask`, `input_ids`, and `token_type_ids` each take 1-1.5 seconds. Assigning `overflow_to_sample_mapping` takes milliseconds, and assigning `offet_mapping` takes 8-9 seconds. It seems that accessing features with more complex structure is taking longer. _I’m very curious what are these assignment statements actually doing, under the hood!_
On a related note, looping over `mapped_ds` takes significantly longer than looping over its features, extracted and zipped:
```python
for x in mapped_ds:
pass
```
takes 15 seconds while
```python
for x in zip(attention_mask, input_ids, offset_mapping, overflow_to_sample_mapping, token_type_ids):
pass
```
takes 1.5 seconds.
**This does have user-facing impact (right?):** If you’re going to loop over the evaluation set in a custom evaluation script (overriding a trainer’s `evaluate` method, say), you’re better off storing the individual features of the evaluation dataset on the trainer instance once and for all and looping over their zip on each evaluation step (of which there can be many!), rather than iterating over the evaluation dataset itself each time. (A custom evaluation script of this sort is used to compute “exact match” and “F1” metrics on the SQuAD data. See the `evaluate` method of `QuestionAnsweringTrainer` in [this file](https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/trainer_qa.py) and the `postprocess_qa_predicitons` function in [this one](https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/utils_qa.py).)
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [November 22, 2021, 11:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-simply-accessing-dataset-features-so-slow%2F12041%2F2 "2021-11-22T11:16:33Z")
Hi,
this behavior is expected.
`dataset[column]` loads the entire dataset column into memory (this is why this part seems so slow to you), so operating directly on this data will be faster than operating on the dataset created with `load_dataset`, which is memory-mapped, i. e., stored in an arrow file by default. However, in many cases, a dataset or dataset transforms are too big to fit in RAM, so having them in memory is not an option and we’d rather pay a penalty (in respect to read/write operations) of having it stored in a file.
If you are sure that you have enough RAM, you can load the dataset as follows to have it in memory:
```python
load_dataset("squad", split="validation", keep_in_memory=True)
```
---
**Author:**  [@mgreenbe](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmgreenbe)%5C
**Post date:** [November 22, 2021, 3:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-simply-accessing-dataset-features-so-slow%2F12041%2F3 "2021-11-22T15:05:22Z")
Makes perfect sense. Thanks! Brief follow-up: When you access a _row_ of the dataset via index, `dataset[i]`, it’s reading from disk unless `keep_in_memory` is set to true on the dataset?
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [November 22, 2021, 3:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-simply-accessing-dataset-features-so-slow%2F12041%2F4 "2021-11-22T15:21:16Z")
> When you access a _row_ of the dataset via index, `dataset[i]`, it’s reading from disk unless `keep_in_memory` is set to true on the dataset?
Yes. You can check at any time whether your dataset is on disk or in memory with `dset.cache_files`, which will return an empty list in the former case.