Zarr

An open-source format for storing large multidimensional arrays in compressed chunks; Diffusion Policy and UMI use it for training data.

Zarr is an open-source, chunked, compressed N-dimensional array storage format, community-maintained with financial support from NumFOCUS, with both v2 and v3 specifications; implementations exist in about ten languages including Python, Rust, C++, and JavaScript, and it was first popular in scientific data fields like climate science and bioimaging. It splits a large array into many small chunks, compressing each one separately, so reading one section only needs to decompress the relevant chunks; the data can live in a local directory, a zip file, or cloud object storage, and supports parallel reads and writes. Conceptually it can be thought of as a “nested dictionary of NumPy arrays,” playing a role close to HDF5, but with a more flexible choice of storage backend, chunking, and compression. In robot learning, the Diffusion Policy codebase uses Zarr to store training data, and UMI follows this same approach, packaging one collection session into a single dataset.zarr.zip file for fast random access during training.

ExampleUMI's example dataset.zarr.zip contains arrays such as camera0_rgb (images sized 2315×224×224×3), robot0_eef_pos (end-effector position), and robot0_gripper_width (gripper opening width), plus an episode_ends array recording which frame each demonstration ends on.

Also called

.zarr.zip

Related

Hierarchical Data Format version 5 · WebDataset · Apache Parquet · LeRobotDataset · Diffusion Policy · Universal Manipulation Interface

Sources

Zarr 官网 (Chinese)
UMI Robot Dataset Community: Zarr data format