Image dataset best practices?

mariosasko
Wed, 16 Feb 2022 15:36:01 +0000

Hi @segments-bert! Thanks, glad you like it.

There is no difference between the dataset.map() approach and this approach, in how the images end up in the parquet files, right?

Yes, there is no difference.

Do I understand correctly that the image bytes are stored in decoded form in the parquet files, i.e. not in their original .jpg or .png format? I assume so, given this comment by @lhoestq .

The image bytes stored in the parquet files are raw/unprocessed.

Would it make sense to add a generic “File” feature, besides the existing Image and Audio features? I’m asking because our data labeling platform also supports video and point cloud labeling, for which the data comes in a variety of file formats (.pcd, .bin, .mp4, …). Would be nice to push such datasets to the hub as well.

An interesting proposal. We will consider it. For now, you can embed file bytes (Value("binary")) in a dataset before push_to_hub and apply some additional preprocessing after loading.