# Datasets - Streaming Output to Arrow?
**URL:**
**Author:**  [@stefanwebb](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fstefanwebb)%5C
**Post date:** [October 21, 2024, 4:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-streaming-output-to-arrow%2F113215%2F1 "2024-10-21T16:41:35Z")
Is there any easy way to stream output to make a data pipeline with Datasets?
The use case is, I’d like to read a HF dataset, run every row through an embedding model, and save the embeddings to disk as a HF dataset.
I wrap the HF Dataset in a Torch Dataloader so I can buffer it with multiple threads and avoid data starving the GPU. Then there’s a Torch inference loop over minibatches that are run through the Sentence Transformers model on the GPU.
The best solution I can see at the moment is to rewrite `ds.save_to_disk` so it runs the Torch inference loop before saving to disk. I can easily create a `pyarrow.parquet.ParquetWriter` to save batches from inference to disk each iteration, but it won’t have the metadata files / convenient sharding for HF. Another method is to store all the Torch tensors in memory and join them with the HF dataset afterwards, which doesn’t work when it doesn’t fit in memory.
Is there a built-in feature or an easier method that I’m missing to accomplish this?
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [October 23, 2024, 2:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-streaming-output-to-arrow%2F113215%2F2 "2024-10-23T14:26:40Z")
Hi ! you can use `Dataset.from_generator()` 🙂
```python
def pipeline():
for batch in dataloader:
for output_example in f(batch):
yield output_example
ds = Dataset.from_generator(pipeline)
# ds.save_to_disk(...)
# or
# ds.push_to_hub(...)
```
---
**Author:**  [@stefanwebb](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fstefanwebb)%5C
**Post date:** [October 23, 2024, 8:42pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-streaming-output-to-arrow%2F113215%2F3 "2024-10-23T20:42:29Z")
Awesome, thanks @lhoestq! That did turn out to be simple…
---
**Author:**  [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [October 24, 2024, 8:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdatasets-streaming-output-to-arrow%2F113215%2F4 "2024-10-24T08:42:40Z")
Closed on October 24, 2024, 8:42am UTC
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.