# Load Dataset and Save as Parquet
**URL:**
**Author:**  [@bkopru](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbkopru)%5C
**Post date:** [June 25, 2024, 1:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-dataset-and-save-as-parquet%2F94003%2F1 "2024-06-25T13:32:00Z")
I am curious whether we can directly save a dataset as parquet as in the original dataset at HF.
For example [fineweb-edu/sample/10BT](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu/tree/main/sample/10BT) has 13 parquet files
```auto
# download the dataset
fw = load_dataset(
"HuggingFaceFW/fineweb-edu",
name="sample-10BT,
split="train",
num_proc=10,
cache_dir=cache_dir,
)
fw.to_parquet(parquet_dir)
```
Loads these files as arrow although they are originally parquet, and when saving with **to\_parquet()** it converts arrow to a single parquet file.
So my questions are:
How can i download/load and save them with the same structure in the repository?
How can i create multiple parquet files instead of single big parquet file with **to\_parquet**?
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [July 10, 2024, 2:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-dataset-and-save-as-parquet%2F94003%2F2 "2024-07-10T14:56:43Z")
Hi ! You can download a repository (and specify a subset of files) using the `huggingface-cli` 🙂
Also right now the `datasets` lib doesn’t support writing to multiple Parquet files yet, though it would be amazing to contribute this feature
Maybe something like this ?
```python
fw.to_parquet("path/to/parquet/", max_shard_size="500MB")
```
---
**Author:**  [@BeeGass](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBeeGass)%5C
**Post date:** [January 6, 2025, 7:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-dataset-and-save-as-parquet%2F94003%2F3 "2025-01-06T19:06:56Z")
By any chance was this made into a feature request or perhaps even implemented? I agree that this would be a great feature
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [January 7, 2025, 11:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-dataset-and-save-as-parquet%2F94003%2F4 "2025-01-07T11:24:31Z")
there is an open issue here: [Save Dataset as Sharded Parquet · Issue #7047 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/issues/7047)