# Possible to stream and create new splits?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpossible-to-stream-and-create-new-splits%2F67214%3E%5C
**Category:** 🤗Datasets\
**Created:** [December 27, 2023, 11:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpossible-to-stream-and-create-new-splits%2F67214 "2023-12-27T23:01:52Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ivnle](https://avatars.discourse-cdn.com/v4/letter/i/b5a626/32.png) [@ivnle](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fivnle)%5C
**Post date:** [December 27, 2023, 11:01pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpossible-to-stream-and-create-new-splits%2F67214%2F1 "2023-12-27T23:01:52Z")

</div>

Say I want to train on a dataset such as [togethercomputer/RedPajama-Data-V2 · Datasets at Hugging Face](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-V2) which only defines a train split. Is it possible to both stream a dataset and create new validation/train splits similar to this example [transformers/examples/pytorch/language-modeling/run\_clm\_no\_trainer.py at main · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm_no_trainer.py) (which does not use streaming)? I’ve copied over the relevant bits.

```auto
    if args.dataset_name is not None:
        # Downloading and loading a dataset from the hub.
        raw_datasets = load_dataset(args.dataset_name, args.dataset_config_name)
        if "validation" not in raw_datasets.keys():
            raw_datasets["validation"] = load_dataset(
                args.dataset_name,
                args.dataset_config_name,
                split=f"train[:{args.validation_split_percentage}%]",
            )
            raw_datasets["train"] = load_dataset(
                args.dataset_name,
                args.dataset_config_name,
                split=f"train[{args.validation_split_percentage}%:]",
            )

```

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [January 4, 2024, 5:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpossible-to-stream-and-create-new-splits%2F67214%2F2 "2024-01-04T17:51:59Z")

</div>

This is not possible for this dataset.

However we plan to make it possible to load percentages of datasets in streaming mode for datasets made of multiples files (shards) and if the number of examples per file is available (this way streaming can skip files until the right percentage of examples is reached)