# Dataset Viewer issue: StreamingRowsError **URL:** **Author:** ![refoundd](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/refoundd/32/34261_2.png) [@refoundd](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frefoundd)%5C **Post date:** [November 9, 2024, 12:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-viewer-issue-streamingrowserror%2F116530%2F1 "2024-11-09T12:48:23Z") The dataset viewer is not working. The image I uploaded has no issues when tested locally, but it returns an error after being uploaded. Is there any code or tool related to packaging images into Parquet files locally? Error details: ```auto Error code: StreamingRowsError Exception: OSError Message: image file is truncated (20 bytes not processed) Traceback: Traceback (most recent call last): File "/src/services/worker/src/worker/utils.py", line 99, in get_rows_or_raise return get_rows( File "/src/libs/libcommon/src/libcommon/utils.py", line 197, in decorator return func(*args, **kwargs) File "/src/services/worker/src/worker/utils.py", line 77, in get_rows rows_plus_one = list(itertools.islice(ds, rows_max_number + 1)) File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 2097, in __iter__ example = _apply_feature_types_on_example( File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 1635, in _apply_feature_types_on_example decoded_example = features.decode_example(encoded_example, token_per_repo_id=token_per_repo_id) File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 2044, in decode_example return { File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 2045, in column_name: decode_nested_example(feature, value, token_per_repo_id=token_per_repo_id) File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/features.py", line 1405, in decode_nested_example return schema.decode_example(obj, token_per_repo_id=token_per_repo_id) File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/features/image.py", line 188, in decode_example image.load() # to avoid "Too many open files" errors File "/src/services/worker/.venv/lib/python3.9/site-packages/PIL/ImageFile.py", line 288, in load raise OSError(msg) OSError: image file is truncated (20 bytes not processed) ``` cc @lhoestq . ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C **Post date:** [November 9, 2024, 12:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-viewer-issue-streamingrowserror%2F116530%2F2 "2024-11-09T12:54:51Z")
Maybe this. > ---
**Author:** ![refoundd](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/refoundd/32/34261_2.png) [@refoundd](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frefoundd)%5C **Post date:** [November 9, 2024, 2:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-viewer-issue-streamingrowserror%2F116530%2F3 "2024-11-09T14:00:42Z")
Thank you for your answer.I hope to display the image on the Dataset Viewer, but it seems that I cannot modify the code of the Dataset Viewer. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C **Post date:** [November 9, 2024, 2:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-viewer-issue-streamingrowserror%2F116530%2F4 "2024-11-09T14:06:28Z")
Dataset Viewer is probably the one built into HF’s GUI. It looks like the only thing to do is to raise an issue on github. > **[GitHub - huggingface/dataset-viewer: Backend that powers the dataset viewer on Hugging...](https://github.com/huggingface/dataset-viewer)** > > Backend that powers the dataset viewer on Hugging Face dataset pages through a public API. > **[Issues · huggingface/dataset-viewer](https://github.com/huggingface/dataset-viewer/issues)** > > Backend that powers the dataset viewer on Hugging Face dataset pages through a public API. - Issues · huggingface/dataset-viewer It seems that you can use the datasets library to convert to parquet on your own. > [@Load Dataset and Save as Parquet](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fload-dataset-and-save-as-parquet%2F94003)%3A > > I am curious whether we can directly save a dataset as parquet as in the original dataset at HF. For example [fineweb-edu/sample/10BT](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu/tree/main/sample/10BT) has 13 parquet files # download the dataset fw = load\_dataset( "HuggingFaceFW/fineweb-edu", name="sample-10BT, split="train", num\_proc=10, cache\_dir=cache\_dir, ) fw.to\_parquet(parquet\_dir) Loads these files as arrow although they are originally parquet, and when saving with to\_parquet() it converts arrow to a single parquet file. So my que… > [@Parquet image dataset](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparquet-image-dataset%2F95641)%3A > > I uploaded a dataset with image and text in a parquet format. The image column is in the format of a dictionary with “bytes” and “path” as required. But the dataset preview cannot recognize the image and display the dictionary instead. How to solve the problem so that images are displayed in the preview section? ---
**Author:** ![refoundd](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/refoundd/32/34261_2.png) [@refoundd](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frefoundd)%5C **Post date:** [November 10, 2024, 12:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-viewer-issue-streamingrowserror%2F116530%2F6 "2024-11-10T12:21:03Z")
Is there any code or tool related to packaging images into Parquet files locally? Thank you. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C **Post date:** [November 10, 2024, 12:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdataset-viewer-issue-streamingrowserror%2F116530%2F7 "2024-11-10T12:39:33Z")
I found it, but it’s still very troublesome🤢, so let’s change our way of thinking. [https://arrow.apache.org/docs/python/parquet.html](https://arrow.apache.org/docs/python/parquet.html) If you open the image file in this way and save it again, I think it will probably be converted to a file format that HF can read automatically. However, be careful, as this will overwrite the file. Try backup whole data set first. ```py def fix_truncated_image(filename): from PIL import ImageFile, Image ImageFile.LOAD_TRUNCATED_IMAGES = True image = Image.open(filename) image.save(filename) fix_truncated_image("data001.jpg") fix_truncated_image("data002.jpg") ``` I think there are a lot of images, so it would be better to automate the process using Python’s **glob** or something similar.