# Compressing, saving, and loading datasets
**URL:**
**Author:**  [@vblagoje](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvblagoje)%5C
**Post date:** [October 30, 2020, 12:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompressing-saving-and-loading-datasets%2F1818%2F1 "2020-10-30T12:15:00Z")
Hello everyone,
I am working with large datasets (Wikipedia), and use map transform to create new datasets. The workflow involves creating new datasets that are saved using save\_to\_disk, and subsequently, I use terminal compression utils to compress the dataset folder. Then I decompress these files and the use load\_from\_disk to load them on other machines. These manual steps are pita.
1. It would be great to use compression within datasets and have one compressed file as a result of save\_to\_disk if so desired.
2. If I could save these datasets to a remote location immediatelly bypassing save\_to\_disk, compression, copy manual steps that would be amazing.
3. Loading these created datasets via URL from s3, gs, etc. via a single load\_dataset call would be a killer.
All the best,
Vladimir
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [October 30, 2020, 2:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompressing-saving-and-loading-datasets%2F1818%2F2 "2020-10-30T14:24:20Z")
I agree that would be super cool to be able to archive and save/load archived dataset from/to a cloud storage. We’re thinking about this actively. Do you think that some dataset versioning logic could be interesting as well ?
---
**Author:**  [@vblagoje](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvblagoje)%5C
**Post date:** [October 30, 2020, 4:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompressing-saving-and-loading-datasets%2F1818%2F3 "2020-10-30T16:09:01Z")
Not important for me. But maybe I am overlooking it. How would you use versioning? Like git for datasets?
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [November 10, 2020, 3:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompressing-saving-and-loading-datasets%2F1818%2F4 "2020-11-10T15:58:34Z")
Yes maybe something similar to git lfs, directly integrated in the library. We’re already doing that for models in the transformers library (since today’s migration).