# Parquet image dataset

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparquet-image-dataset%2F95641%3E%5C
**Category:** 🤗Datasets\
**Created:** [July 5, 2024, 7:03am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparquet-image-dataset%2F95641 "2024-07-05T07:03:41Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![LouisChen15](https://avatars.discourse-cdn.com/v4/letter/l/7feea3/32.png) [@LouisChen15](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLouisChen15)%5C
**Post date:** [July 5, 2024, 7:03am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparquet-image-dataset%2F95641%2F1 "2024-07-05T07:03:41Z")

</div>

I uploaded a dataset with image and text in a parquet format. The image column is in the format of a dictionary with “bytes” and “path” as required. But the dataset preview cannot recognize the image and display the dictionary instead. How to solve the problem so that images are displayed in the preview section?

---

<div class="post-metadata">

**Author:** ![severo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/severo/32/27449_2.png) [@severo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsevero)%5C
**Post date:** [July 8, 2024, 8:41am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparquet-image-dataset%2F95641%2F2 "2024-07-08T08:41:46Z")

</div>

can you share the repository?

---

<div class="post-metadata">

**Author:** ![LouisChen15](https://avatars.discourse-cdn.com/v4/letter/l/7feea3/32.png) [@LouisChen15](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLouisChen15)%5C
**Post date:** [July 8, 2024, 6:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparquet-image-dataset%2F95641%2F3 "2024-07-08T18:45:26Z")

</div>

Thanks for replying me. The dataset is this: [LouisChen15/ConstructionSite · Datasets at Hugging Face](https://huggingface.co/datasets/LouisChen15/ConstructionSite). I removed the original dataset and only left ten images as a demo. But they have the same issue as described.

---

<div class="post-metadata">

**Author:** ![severo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/severo/32/27449_2.png) [@severo](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsevero)%5C
**Post date:** [July 8, 2024, 7:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparquet-image-dataset%2F95641%2F4 "2024-07-08T19:32:17Z")

</div>

cc @lhoestq? Images in a parquet file.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [July 9, 2024, 1:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparquet-image-dataset%2F95641%2F5 "2024-07-09T13:37:34Z")

</div>

Hi ! Pandas sets the type of the “image” column to be a struct of bytes and path, since it doesn’t have an image type (yet ?)

Ideally it would be great to have a way to define types metadata, maybe we could have something like this in the future ?

```python
df.image.attrs = {"dtype": "image"}
df.to_parquet("hf://datasets/LouisChen15/ConstructionSite/test_split_demo.parquet")

```

Anyway, right now if you want to set the type to image, you can define the types in the README.md in YAML:

```yaml
dataset_info:
  features:
  - name: image  
    dtype: image
  - name: image_id
    dtype: string
  - name: image_caption
    dtype: string
  - name: illumination
    dtype: string
  - name: camera_distance
    dtype: string
  - name: view
    dtype: string
  - name: quality_of_info
    dtype: string
  - name: rule_1_violation
    struct:
    - name: bounding_box
      sequence:
        sequence: float64
    - name: reason
      dtype: string
  - name: rule_2_violation
    dtype: 'null'
  - name: rule_3_violation
    struct:
    - name: bounding_box
      sequence:
        sequence: float64
    - name: reason
      dtype: string
  - name: rule_4_violation
    dtype: 'null'
  - name: excavator
    sequence:
      sequence: float64
  - name: rebar
    sequence:
      sequence: float64
  - name: worker_with_white_hard_hat
    sequence: 'null'

```

---

<div class="post-metadata">

**Author:** ![LouisChen15](https://avatars.discourse-cdn.com/v4/letter/l/7feea3/32.png) [@LouisChen15](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FLouisChen15)%5C
**Post date:** [July 9, 2024, 8:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparquet-image-dataset%2F95641%2F7 "2024-07-09T20:36:21Z")

</div>

Thank you, it finally works, the metadata is exactly what I need. However, I did try using metadata before to solve the problem, but I only include the image part:

```auto
-name: image
 dtype: image

```

But it did not work. Does it mean I have to correctly specify the data type of all the “names” so that the system can process it?

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [July 10, 2024, 2:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparquet-image-dataset%2F95641%2F8 "2024-07-10T14:38:02Z")

</div>

Yea all the columns/types are required in the YAML
