Image dataset best practices?

segments-bert
Fri, 21 Jan 2022 11:54:09 +0000

Hi! I’m one of the founders of Segments.ai, a data labeling platform for computer vision. We’re working on an integration with HuggingFace, making it possible to export labeled datasets to the hub.

From reading the docs and toying around a bit, I found there’s a few potential ways to set up an image dataset:

1. Keep the image files out of the repository, and download them from their URLs (they’re hosted in the cloud) in the dataset loading script. The disadvantage here is that if the image URLs ever become unavailable, the dataset also won’t work anymore.

2. Store the image files in the repository, packed together in a few large parquet files, using git-lfs. This is basically what happens when you create a dataset with an image column locally, and run dataset.push_to_hub(). See this dataset.

3. Store the image files in the repository as individual jpg/png files, using git-lfs. Compared to the previous approach, this requires a custom dataset loading script. This seems cleaner from a versioning point of view: when images are added or removed later on, it leads to a compact diff compared to working with the parquet files. But perhaps it’s not ideal to have so many small files in a git-lfs repo.

Do you have any recommendations on what would be the cleanest approach that is considered best practice?