# Streaming for Saving
**URL:**
**Author:**  [@sinamoeini](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsinamoeini)%5C
**Post date:** [January 26, 2025, 10:30am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-for-saving%2F137377%2F1 "2025-01-26T10:30:59Z")
Hi,
I am looking for a way to download a large dataset, transform it and then upload it to another location. Note that the transformations for each instance is independent of others.
I can load the dataset in streaming mode and start the transformation but cannot find a way to write to huggingface hub (in batches) in as download and transformation are ongoing. Wondering if such pattern exists
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [January 26, 2025, 11:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-for-saving%2F137377%2F2 "2025-01-26T11:47:44Z")
In the case of datasets library’s push\_to\_hub, I think you couldn’t upload the data unless all of it was available…
If the files are outputted frequently, in the worst case, there is a way to manually upload them one after another using **HfApi** …
> **[HfApi Client](https://huggingface.co/docs/huggingface_hub/v0.27.1/en/package_reference/hf_api#huggingface_hub.HfApi.upload_folder)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
---
**Author:**  [@Zoe0427](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FZoe0427)%5C
**Post date:** [September 12, 2025, 6:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-for-saving%2F137377%2F3 "2025-09-12T18:56:05Z")
What if I just want to transform the dataset and then save in a streaming way? If the dataset is large, the CPU occupied memory become larger when transfering the dataset. Or should I transfer and save the data in parts instead of waiting until the entire dataset has been transferred?
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [September 12, 2025, 11:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fstreaming-for-saving%2F137377%2F4 "2025-09-12T23:37:04Z")
Yeah. It is now possible to save parquet files per shard or [upload them incrementally](https://huggingface.co/docs/datasets/en/stream#save).