# Slow Iteration speed (with and without keep\_in\_memory=True)
**URL:**
**Author:**  [@DanielNlp](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDanielNlp)%5C
**Post date:** [March 11, 2023, 3:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fslow-iteration-speed-with-and-without-keep-in-memory-true%2F33587%2F1 "2023-03-11T15:50:20Z")
I have a dataset containing around 1.7 entries and the following structure:
ColA → tokenized text
ColB → List(3 tokenized text samples)
I was comparing different methods of iterating over the dataset using a torch DataLoader and found the HF datasets to be consistently slower.
All versions use the same collate\_fn
```auto
# mydata = 100k samples
#A -> ~81sec
dataset = load_from_disk("mydata",keep_in_memory=True)
loader = DataLoader(dataset, batch_size=1000, collate_fn = collate_fn)
#B -> ~83sec
dataset = load_from_disk("mydata",keep_in_memory=False)
loader = DataLoader(dataset, batch_size=1000, collate_fn = collate_fn)
#C -> ~45sec
from torch.utils.data import Dataset, DataLoader
data= pd.read_parquet("mydata")
class MyDataset(Dataset):
def __init__ (self, pd_dataset):
self.data = pd_dataset
def __len__ (self):
return len(self.data)
def __getitem__ (self, idx):
return {"cola":self.data.cola[idx], "colb":self.data.colb[idx]}
m =MyDataset(data)
loader = DataLoader(m, batch_size=1000, collate_fn = collate_fn)
#Testing
start_time = time.time()
for l in loader:
pass
print("--- %s seconds ---" % (time.time() - start_time))
```
I am aware that the results may not be as accurate as possible but the differences should give some indication.
Questions:
- Why is my pandas version significantly faster than iterating over the hugginface datasets?
- Why is there no significant performance difference between keep\_in\_memory True/False?
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [March 13, 2023, 4:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fslow-iteration-speed-with-and-without-keep-in-memory-true%2F33587%2F2 "2023-03-13T16:02:01Z")
To read tokenized text from Arrow, the bottleneck is often the conversion of the tokenized Arrow data to pythons lists.
It’s much faster to load them as torch tensors directly - since the data is loaded using zero-copy from your disk:
```python
dataset = load_from_disk("mydata",keep_in_memory=False)
dataset = dataset.with_format("torch")
loader = DataLoader(dataset, batch_size=1000, collate_fn = collate_fn)
```
Also make sure to use the latest versions of `datasets` and `torch` (minimum 2.10 and 1.13 to get the best speed)
---
**Author:**  [@DanielNlp](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDanielNlp)%5C
**Post date:** [March 13, 2023, 9:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fslow-iteration-speed-with-and-without-keep-in-memory-true%2F33587%2F3 "2023-03-13T21:17:37Z")
Thank you for the reply!
I should add that the data in my dataset is tokenized but not truncated as I want to perform dynamic padding on the batches. Thus I can not convert the dataset into torch tensors of the same size. I found that in this case, the with\_format(“torch”) reduces the speed.
Any suggestions on how to still improve performance?
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [March 14, 2023, 3:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fslow-iteration-speed-with-and-without-keep-in-memory-true%2F33587%2F4 "2023-03-14T15:06:23Z")
> I found that in this case, the with\_format(“torch”) reduces the speed.
Oh this shouldn’t be the case, let me know if you find what’s slowing things down for you (using a profiler or stopping the iteration in the middle to check the traceback)