co25-04-agrinext — SwissCrop25 storage-layout experiments

The same Sentinel-2 minicubes stored three ways, so the AgriNext work can pick a layout on evidence. The pixels are identical; what differs is the container, and with it the object count and what a client needs in order to read a cube at all.

All three derive from one shard of EOA-team/SwissCrop25, sentinel2/2019/2019_001.tar: 243 tiles, 128 × 128 px at 10 m, EPSG:32632, 2019. Each tile holds 24 arrays — 21 data variables plus time/lat/lon — chunked (time, 128, 64), uint16, blosc/zstd-3.

001-SwissCrop25/sentinel2/2019/2019_001.tar       the shard as published
001-SwissCrop25/sentinel2/2019.json               its Kerchunk index
002-SwissCrop25/sentinel2/2019/S2_*.zarr/         243 plain Zarr v2 stores
003-SwissCrop25/sentinel2/2019/S2_*.zarr/         243 sharded Zarr v3 stores

Object count is the axis: 001 is 2 objects but unreadable without its sidecar index; 002 opens directly at the cost of 22k objects; 003 keeps that direct readability and roughly halves the count by sharding.

Measured from a browser client (WASM blosc decode, local server, so transfer is loopback): one date of a 3 × 3 tile RGB view costs 69 MB and ~1.5 s in every layout. The chunking decides that, not the container — a chunk holds the whole year, so one date transfers all 143 timesteps. 003 issues more requests than 002 for the same read (81 vs 54), trading a shard-index round trip per array for half the objects at rest. See code/benchmark/ in the workspace repo.

002 — read a cube directly

No Kerchunk, no reference file, no download step:

import os, xarray as xr

ds = xr.open_zarr(
    "hf://buckets/SDSC/co25-04-agrinext/002-SwissCrop25/sentinel2/2019/"
    "S2_262620_5110700_20190103_20191231.zarr",
    storage_options={"token": os.environ["HF_TOKEN"]},
)
# time: 143, lat: 128, lon: 128 · 21 data variables

Tile names are S2_<easting>_<northing>_<start>_<end>.zarr. Consolidated .zmetadata ships with every store, so opening costs one request rather than ~48. hf:// paths need huggingface_hub (which registers the fsspec hf protocol); the S3 gateway at https://s3.hf.co/SDSC serves the same objects to boto3, rclone and s5cmd using an HFAK… keypair.

003 — read the Zarr v3 copy

Same call, but v3 carries dimension names natively, so xarray needs no hints:

ds = xr.open_zarr(
    "hf://buckets/SDSC/co25-04-agrinext/003-SwissCrop25/sentinel2/2019/"
    "S2_262620_5110700_20190103_20191231.zarr",
    storage_options={"token": os.environ["HF_TOKEN"]},
)

Requires zarr >= 3.0. Each array is one shard, so a reader fetches the shard index before the chunk — one extra round trip per array, in exchange for 49 files per tile instead of 91.

001 — read the packed shard

The tar holds 243 .zarr.zip members, so nothing opens a cube directly. The Kerchunk file supplies (offset, length) per zarr key and presents the tar as a virtual store. It works because the zip members are stored uncompressed (compress_type=0), so raw byte ranges land exactly on zarr chunks.

# note the group= : the reference file's root group is empty, one subgroup per tile
ds = xr.open_dataset("reference://", engine="zarr",
                     group="S2_262620_5110700_20190103_20191231.zarr",
                     backend_kwargs={"consolidated": False,
                                     "storage_options": {"fo": "2019.json",
                                                         "remote_protocol": "file"}})

Keep this as the compact archival copy, faithful to upstream.

How 002 was produced

Both containers unpacked — a pure byte copy, since chunks stay blosc-compressed exactly as written. Nothing is decoded or recompressed.

Every one of the 22,113 objects was compared against the byte range Kerchunk claims for it in 2019_001.tar: all identical. Reading tile S2_262620_5110700 returns s2_B04[0].mean() = 550.67041015625 through every route — Kerchunk over the tar, the .zarr.zip, the extracted store locally, and the same store over hf://.

003-SwissCrop25 is that same data converted 1:1 to Zarr v3 with sharding — 49 files per tile instead of 91, chunking unchanged. It needs zarr >= 3.0; v2-only readers cannot open it. See its own README.md for the two intentional metadata differences (fill_value: null cannot survive into v3, and _ARRAY_DIMENSIONS is promoted to dimension_names).

Data caveats

Inherited from upstream, true of all three layouts:

• lat/lon are northing/easting in metres on EPSG:32632, not degrees.
• Coordinates label upper-left pixel corners and lat descends, so the extent runs one pixel past the last label.
• Time is ragged: 133 tiles have 143 timesteps, 110 have 144. Concatenating tiles along a shared time axis requires aligning or padding first.
• Fill value is 65535; s2_mask is a deep-learning cloud mask.
• 2019.json is also present under 002-SwissCrop25/, where it is redundant — it indexes the tar, not the extracted stores.

Cubes were built with the EOA minicuber from Microsoft Planetary Computer data (baseline correction, cloud masking, BRDF correction, reprojection, regridding). That history is preserved in the root attributes of every store.

Licence follows the upstream dataset.