# Best practice loading images files
**URL:**
**Author:**  [@phihung](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fphihung)%5C
**Post date:** [March 25, 2022, 3:54pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-practice-loading-images-files%2F16081%2F1 "2022-03-25T15:54:52Z")
Hi everybody,
I’m looking for best practices for loading image files.
I’ve tried so far
```python
ds = Dataset.from_dict({"path": [my_jpg]})
# Method 1: cast + transform
ds_1 = (
ds.rename_column("path", "pixel_values")
.cast_column("pixel_values", Image())
.with_transform(transform_to_tensor)
)
# Method 2: map + map
features = Features({"pixel_values": Array3D(..., dtype="float64")})
ds_2 = (
ds.map(read_images, batched=True, batch_size=64)
.map(transform_to_numpy, features=features, batched=True, batch_size=64, num_proc=8)
.with_format("pt")
)
# Method 3: map + transform
ds_3 = (
ds.map(read_images, batched=True, batch_size=64)
.with_transform(transform_to_tensor)
)
# Method 4: ONE MAP
ds_4 = (
ds.map(read_images_and_process_to_numpy, batched=True, batch_size=64)
.with_format("pt")
)
```
- Method 1 is really nice but as everything is lazy loading, the training loop is terribly slow (time spending on disk + CPU data loading)
- Method 2 and 3: 1 hour for the read\_images map with 8Gb data
- Method 4: kind of working. But maybe not the most elegant? All data aug is done during loading.
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [March 26, 2022, 12:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-practice-loading-images-files%2F16081%2F2 "2022-03-26T12:03:56Z")
Hi,
Probably the easiest way to load images into a Dataset is by leveraging the new [ImageFolder](https://huggingface.co/docs/datasets/v2.0.0/en/image_process#imagefolder) builder.
---
**Author:**  [@lhoestq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flhoestq)%5C
**Post date:** [April 7, 2022, 9:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-practice-loading-images-files%2F16081%2F3 "2022-04-07T09:50:41Z")
method 1 saves you some disk space, since it reads the image from the path you provided without copying the image on your disk
method 2 decodes your image completely and saves it on disk. The Array3D type makes it very fast to read the images, with zero-copy reads of the arrays from your disk. So you get high throughput at the cost of disk space.
method 3 doesn’t give better performance than 1, but it requires some processing and your map call copies the image in a new dataset, which consumes a bit of disk space
method 4 is same as 2, but since it doesn’t have any intermediate step you save some disk space. Note that you didn’t use Array3D for this one, but you probably should to get the best performance 😉
As mentioned above, feel free to use the [imagefolder](https://huggingface.co/docs/datasets/image_process#imagefolder) data loader, and combined with `with_transform(transform_to_tensor)` you get a good tradeoff in performance vs disk space used
> the training loop is terribly slow (time spending on disk + CPU data loading)
To improve data loading speed, feel free to use a PyToch DataLoader with `num_workers` \> 1, this way the image decoding can be done in parallel in subprocesses.
---
**Author:**  [@omgwenxx](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fomgwenxx)%5C
**Post date:** [March 27, 2024, 10:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbest-practice-loading-images-files%2F16081%2F4 "2024-03-27T10:55:59Z")
Took me way too long to load 30GB of images with the ImageFolder, used [this](https://ngwaifoong92.medium.com/how-to-speed-up-data-loading-for-machine-learning-training-d10d202d5d13) approach instead which decreased 70 seconds loading into 0.03 seconds.