# Handle errors when loading images (404, corrupted, etc)

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhandle-errors-when-loading-images-404-corrupted-etc%2F50318%3E%5C
**Category:** 🤗Datasets\
**Created:** [August 10, 2023, 6:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhandle-errors-when-loading-images-404-corrupted-etc%2F50318 "2023-08-10T18:08:05Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![nlassaux](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nlassaux/32/32462_2.png) [@nlassaux](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnlassaux)%5C
**Post date:** [August 10, 2023, 6:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhandle-errors-when-loading-images-404-corrupted-etc%2F50318%2F1 "2023-08-10T18:08:05Z")

</div>

Hello, I am loading images into a Dataset, by casting their urls as `datasets.Image` objects.

```auto
def load_dataset(db_client: DBClient) -> Dataset:
    """Loads the dataset from the given bucket."""
    paths = db_client.missing_image_paths()
    paths = list(paths)

    def url_from_path(path: str) -> str:
        return f'gs://{BUCKET}/{FOLDER}{path}'
    
    return Dataset.from_dict({
        'image': [url_from_path(path) for path in paths],
        'filename': paths
    }).cast_column('image', Image())

```

Now, some of these images don’t exist anymore. So with `print(dataset[0])`, I get:  
`{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=1920x1280 at 0x132D99190>, 'filename': 'b57ed2793e6a8ae06382c78a87863b8d.jpg'}` ✅

But if I try to load more, at some point, I get a message similar to that: `PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x13592fab0>` 🚫

Is there some way to specify that we want to ignore those issues and discard the images when that happens?

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [August 16, 2023, 3:35pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhandle-errors-when-loading-images-404-corrupted-etc%2F50318%2F2 "2023-08-16T15:35:16Z")

</div>

Hi! You can remove invalid image files with

```python
dataset = dataset.cast_column("image", datasets.Image(decode=False))

def has_valid_image(ex):
    try:
        PIL.Image.open(ex["image"]["path"])
    except Exception:
        return False
    return True

dataset = dataset.filter(has_valid_image)
dataset = dataset.cast_column("image", datasets.Image(decode=True))

```

---

<div class="post-metadata">

**Author:** ![nlassaux](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nlassaux/32/32462_2.png) [@nlassaux](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnlassaux)%5C
**Post date:** [August 16, 2023, 3:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhandle-errors-when-loading-images-404-corrupted-etc%2F50318%2F3 "2023-08-16T15:37:37Z")

</div>

Hi @mariosasko, and thanks. It would work, but require to load images twice. I wonder if there is any way to pass on the loading on the fly instead, to divide the amount of work by 2.

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmariosasko)%5C
**Post date:** [August 16, 2023, 4:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhandle-errors-when-loading-images-404-corrupted-etc%2F50318%2F4 "2023-08-16T16:15:56Z")

</div>

This is the lazy approach

```python
dataset = dataset.cast_column("image", datasets.Image(decode=False))

def invalid_images_as_none(batch):
    images = []
    for image in batch["image"]:
       try:
           image = PIL.Image.open(image["path"])
       except Exception:
           image = None
       images.append(image)
    batch["image"] = image
    return batch

dataset = dataset.with_transform(invalid_images_as_none)

```

---

<div class="post-metadata">

**Author:** ![nlassaux](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nlassaux/32/32462_2.png) [@nlassaux](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnlassaux)%5C
**Post date:** [August 17, 2023, 8:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhandle-errors-when-loading-images-404-corrupted-etc%2F50318%2F5 "2023-08-17T08:50:33Z")

</div>

Thank you! @mariosasko