# Loading just part of dataset
**URL:**
**Author:**  [@majdoddin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmajdoddin)%5C
**Post date:** [May 2, 2023, 7:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-just-part-of-dataset%2F38463%2F1 "2023-05-02T19:46:33Z")
Some datasets are huge, which makes it impractical to load all of it from Hf with load\_dataset(), when debugging the code. Therefore one needs just load part of the dataset, say the first 10k rows. But how?
I know it is possible to load a part of dataset to memory with [“slice splitting”](https://huggingface.co/docs/datasets/v2.12.0/en/loading#slice-splits), but it appears that it first downloads the whole dataset if it is not cached.
---
**Author:**  [@stevhliu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fstevhliu)%5C
**Post date:** [May 2, 2023, 8:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-just-part-of-dataset%2F38463%2F2 "2023-05-02T20:31:40Z")
You can [stream](https://huggingface.co/docs/datasets/stream) the dataset which doesn’t download anything, and lets you use it instantly 🙂
---
**Author:**  [@majdoddin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmajdoddin)%5C
**Post date:** [May 2, 2023, 8:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-just-part-of-dataset%2F38463%2F3 "2023-05-02T20:59:49Z")
Thank you very much, That’s it. Great!
---
**Author:**  [@pbk0](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpbk0)%5C
**Post date:** [May 5, 2024, 3:22am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-just-part-of-dataset%2F38463%2F4 "2024-05-05T03:22:15Z")
Yes, but you still use excess download bandwidth and then slice it on your machine in memory.
---
**Author:**  [@neonwatty](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fneonwatty)%5C
**Post date:** [February 25, 2025, 2:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floading-just-part-of-dataset%2F38463%2F5 "2025-02-25T14:48:34Z")
Faced this issue as well so wrote a short script that pulls a hub dataset, creates a small sample of it, and pushes the sample data to the hub as a new dataset.
```python
def create_sample_dataset(full_dataset_name, sample_count=100, username="my-username", cache_dir="./dataset"):
# Create a directory to save the sampled dataset
os.makedirs(cache_dir, exist_ok=True)
# Get the dataset name
dataset_name = full_dataset_name.split("/")[-1]
dataset_name_sample = f"{dataset_name}-sample-{sample_count}"
# Load the dataset
dataset = datasets.load_dataset(full_dataset_name, cache_dir=cache_dir)
# Sample 100 rows from the training split (or modify for other splits)
train_sample = dataset["train"].shuffle(seed=42).select(range(sample_count))
test_sample = dataset["test"].shuffle(seed=42).select(range(sample_count))
# Push to hub
train_sample.push_to_hub(dataset_name_sample, split="train")
print("INFO: Train split pushed to the hub successfully")
test_sample.push_to_hub(dataset_name_sample, split="test")
print("INFO: Test split pushed to the hub successfully")
```
Once sampled / pushed you have a smaller version of your dataset in the hub to pull from.
The [full gist is here](https://gist.github.com/neonwatty/2205277501928f5945726e9e90950ae9).