# Skipping in Steaming mode takes forever
**URL:**
**Author:**  [@Shaier](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FShaier)%5C
**Post date:** [April 25, 2023, 9:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fskipping-in-steaming-mode-takes-forever%2F37718%2F1 "2023-04-25T21:03:07Z")
I’m trying to use a large dataset (30M+ entries) in streaming mode with skipping the first 12M lines:
`full_pubmed = load_dataset('pubmed', streaming=True)['train'].skip(12000000)`
But it’s been over 10 min for the following code to run
`list(full_pubmed.take(20))`.
What is the point of “skip” if it just iterates lines one by one (which is why I assume it takes so much time)? Unless I’m missing something and there’s a better way to skip entries?
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [April 26, 2023, 4:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fskipping-in-steaming-mode-takes-forever%2F37718%2F2 "2023-04-26T16:57:54Z")
Hi ! The orignal pubmed data files are around 1k xml files and we can’t really know in advance where the example at position 12M is located unfortunately, so it has to iterate on the all the examples before finding it.
At one point we’ll support fast skipping if the dataset is made of supported data files like parquet, for which we know the length in advance
---
**Author:**  [@Shaier](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FShaier)%5C
**Post date:** [April 28, 2023, 4:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fskipping-in-steaming-mode-takes-forever%2F37718%2F3 "2023-04-28T04:16:19Z")
Got it! Thanks a lot for the help
---
**Author:**  [@samsja](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsamsja)%5C
**Post date:** [May 7, 2024, 9:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fskipping-in-steaming-mode-takes-forever%2F37718%2F4 "2024-05-07T09:56:26Z")
Hey any progress on this topic ? Thanks in advance
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [May 13, 2024, 4:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fskipping-in-steaming-mode-takes-forever%2F37718%2F5 "2024-05-13T16:23:13Z")
Not yet, though it would be cool to let the ExamplesIterable (see [datasets/src/datasets/iterable\_dataset.py at main · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/blob/main/src/datasets/iterable_dataset.py)) have a length that can be used to know how many of them should be skipped when calling `.skip()` with a high value