# Parquet compression for image dataset
**URL:**
**Author:**  [@Divelix](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDivelix)%5C
**Post date:** [December 4, 2023, 11:42am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparquet-compression-for-image-dataset%2F64514%2F1 "2023-12-04T11:42:49Z")
Hi, Iâm new to hf and recently, while dealing with [sidewalk-semantic](https://huggingface.co/datasets/segments/sidewalk-semantic) dataset, I found out that its parquet version has incredible compression rate (raw images are ~2.4 Gb, while parquet is 324 Mb). Also I found [their post on this forum](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fimage-dataset-best-practices%2F13974)%2C but it didât help much. So I tried to compress images into parquet by myself, but got increased size, instead of reduced size.
There is my code:
```python
from pathlib import Path
from datasets import Dataset
from PIL import Image
if __name__ == " __main__":
workdir = Path("/path/to/data")
images = workdir / "images"
labels = workdir / "labels"
out_name = "0000.parquet"
out_path = workdir / "my_data.parquet"
data = {
"images": [
Image.open(images / "000000.png").tobytes(),
Image.open(images / "000001.png").tobytes(),
],
"labels": [
Image.open(labels / "000000.png").tobytes(),
Image.open(labels / "000001.png").tobytes(),
],
}
dataset = Dataset.from_dict(data)
dataset.to_parquet(out_path)
```
There I decompressed PNG images into raw bytes and packed into parquet, willing `to_parquet()` function would apply its own compression, but it turned out Iâm doing something wrong (resulting parquet is bigger than source images). Guide me please to right direction.
**My goal:** compress PNG images on disk into parquet with reduced size (like in sidewalk-semantic) locally (without `push_to_hub`).
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [December 4, 2023, 3:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparquet-compression-for-image-dataset%2F64514%2F2 "2023-12-04T15:17:34Z")
You can open a discussion in the repo to ask the authors this question.
How did you get the size of the âraw imagesâ? Parquet usually does not compress raw bytes (images) efficiently, but we still use it for images to be consistent with the other data types. Instead, you should use a different image format (or the `compress_level` in `PIL.Image.save`) to reduce the imagesâ size.
---
**Author:**  [@Divelix](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDivelix)%5C
**Post date:** [December 4, 2023, 4:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparquet-compression-for-image-dataset%2F64514%2F3 "2023-12-04T16:08:17Z")
Hi @mariosasko, thank you for feedback! About âraw imagesâ: I took dataset field âpixel\_valuesâ (bytes type) and decoded image from them via `Image.open()`. Resulting folder weights ~2.4 Gb (~2.4 Mb per image \* 1000). I understand your suggestion about compress\_level, but those one used in the dataset seems like lossless one, which is really interesting.
---
**Author:**  [@Divelix](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDivelix)%5C
**Post date:** [December 4, 2023, 4:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparquet-compression-for-image-dataset%2F64514%2F4 "2023-12-04T16:20:23Z")
I followed your advice and started discussion on dataset page, but the reason I didnât do that earlier is that from forum discussion linked above I concluded that compression happens inside `push_to_hub()` method of `dataset` library and not specific to `sidewalk-semantic` dataset.
---
**Author:**  [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [December 5, 2023, 2:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparquet-compression-for-image-dataset%2F64514%2F5 "2023-12-05T14:43:26Z")
310 MB is the size I get when I iterate over the dataset and save its images to an image folder: [Google Colab](https://colab.research.google.com/drive/1p3DGIdyyFGMHMld4XAK2967_UHFqqveO?usp=sharing).
---
**Author:**  [@Divelix](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FDivelix)%5C
**Post date:** [December 7, 2023, 4:27pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparquet-compression-for-image-dataset%2F64514%2F6 "2023-12-07T16:27:14Z")
Ok, I see that you extracted âpixel\_valuesâ as JPG and âlabelâ as PNG, while I extracted both as PNG. So impressive compression rate they achieved is just plain jpeg đ. Thank you for help.