Measuring memory usage of Zarr array storage operations using memray.
In an ideal world array storage operations would be zero-copy, but many libraries do not achieve this in practice. The scripts here measure what the actual empirical behaviour is across different filesystems (local/cloud), Zarr stores (local/s3fs/obstore/zarrs), compression settings (using numcodecs), Zarr implementations (Zarr Python v2/v3, and zarrs via Zarrista), and Zarr formats (2/3).
Buffer.to_bytes()) before handing an encoded chunk to Rust. Icechunk writes are now best achievable, 0 copies uncompressed and 1 compressed. Reads are unchanged, see https://github.com/earth-mover/icechunk/issues/2345pipe avoid a copy by using cramjam: https://github.com/fsspec/s3fs/issues/959#issuecomment-2829560035. Path forward for Zarr is not clear though.The workload is simple: create a random 100MB NumPy array and write it to Zarr storage in a single chunk. Then (in a separate process) read it back from storage into a new NumPy array.
Local filesystem with current library versions. S3 is not tracked yet, but for obstore the numbers for S3 and local should be the same.
These tables are regenerated by make. The Libraries column reports the versions actually exercised: configs/ pins zarr and zarrista but leaves the store libraries floating, so each profile records its own versions in a profiles/*.versions.json sidecar as it runs, and the table is built from those.
zarrista appears in the reads table but not the writes table for icechunk: it can read an icechunk repository, but not write to one yet (its session bridge reconstructs a separate session internally, so writes are silently lost). Those reads are of data written by the zarr-python + icechunk target.
Number of extra copies needed to write an array to storage using Zarr. (Links are to memray flamegraphs. Bold indicates best achievable.)
| Filesystem | Store | Libraries | Zarr format | Uncompressed | Compressed |
|---|---|---|---|---|---|
| Local | local | zarr 3.3.0 | 3 | 0 | 1 |
| obstore | zarr 3.3.0, obstore 0.11.1 | 3 | 0 | 1 | |
| icechunk | zarr 3.3.0, icechunk 2.2.0 | 3 | 0 | 1 | |
| local | zarrista 0.1.0 | 3 | 1 | 1 | |
| obstore | zarrista 0.1.0, obstore 0.11.1 | 3 | 1 | 1 |
Number of extra copies needed to read an array from storage using Zarr. (Links are to memray flamegraphs. Bold indicates best achievable.)
| Filesystem | Store | Libraries | Zarr format | Uncompressed | Compressed |
|---|---|---|---|---|---|
| Local | local | zarr 3.3.0 | 3 | 1 | 2 |
| obstore | zarr 3.3.0, obstore 0.11.1 | 3 | 1 | 2 | |
| icechunk | zarr 3.3.0, icechunk 2.2.0 | 3 | 2 | 2 | |
| local | zarrista 0.1.0 | 3 | 0 | 1 | |
| obstore | zarrista 0.1.0, obstore 0.11.1 | 3 | 0 | 1 | |
| icechunk | zarrista 0.1.0, icechunk 2.2.0 | 3 | 1 | 1 |
Kept for historical interest: older zarr-python versions and S3. These are frozen and not regenerated - as new library versions are released, the current rows above are promoted here by hand. These rows predate the version sidecars, so they name only the zarr and numcodecs versions that were recorded at the time.
| Filesystem | Store | Libraries | Zarr format | Uncompressed | Compressed |
|---|---|---|---|---|---|
| Local | local | zarr 2.18.7, numcodecs 0.15.1 | 2 | 0 | 2 |
| zarr 3.0.6, numcodecs 0.15.1 | 3 | 1 | 2 | ||
| zarr 3.0.8, numcodecs 0.16.1 | 3 | 0 | 1 | ||
| obstore | zarr 3.0.8, numcodecs 0.16.1 | 3 | 0 | 1 | |
| S3 | s3fs | zarr 2.18.7, numcodecs 0.15.1 | 2 | 1 | 2 |
| zarr 3.0.6, numcodecs 0.15.1 | 3 | 1 | 2 | ||
| obstore | zarr 3.0.8, numcodecs 0.16.1 | 3 | 0 | 1 |
| Filesystem | Store | Libraries | Zarr format | Uncompressed | Compressed |
|---|---|---|---|---|---|
| Local | local | zarr 2.18.7, numcodecs 0.15.1 | 2 | 1 | 1 |
| zarr 3.0.6, numcodecs 0.15.1 | 3 | 1 | 2 | ||
| obstore | zarr 3.0.8, numcodecs 0.16.1 | 3 | 1 | 2 | |
| S3 | s3fs | zarr 2.18.7, numcodecs 0.15.1 | 2 | 2 | 2 |
| zarr 3.0.6, numcodecs 0.15.1 | 3 | 2 | 2 | ||
| obstore | zarr 3.0.8, numcodecs 0.16.1 | 3 | 1 | 2 |
Update: some of these paths have been fixed now, so the below may not represent the code in the latest releases.
This delves into what is happening for the different code paths, and suggests some remedies to reduce the number of buffer copies.
write() method (in DirectoryStore), and since arrays implement the buffer protocol, no copy is made.maybe_convert, called from FSMap.setitems()): bytes(memoryview(value)).LocalStore): memoryview(value.as_numpy_array().tobytes()). A similar thing happens in FsspecStore and obstore.value Buffer is exposed via the buffer protocol without making a copy._process_chunk(). So there is necessarily a buffer copy, since the bytes are never read directly into the output array.CodecPipeline has a read() method that separates reading the bytes from storage, and filling the output array (just like v2). The ByteGetter class has no way of reading directly into an output array.Buffer.as_buffer_like method - https://github.com/zarr-developers/zarr-python/issues/2925 (fixed)Buffers to memoryviews - https://github.com/zarr-developers/numcodecs/pull/656 (fixed)pipe - https://github.com/fsspec/s3fs/issues/959Requires uv. Per-configuration dependencies are declared in configs/. uv resolves and caches each environment automatically - no manual pip install or virtual env activation is needed.
The actively tracked matrix (local filesystem, latest zarr-python, and zarrista):
make current-profiles
To run a single profile:
make profiles/write-local-zarr-3.3.0-fsspec-compressed.bin
For read profiles, the corresponding write is run first automatically (the data must exist).
When a new zarr-python is released, bump CURRENT_ZARR in the Makefile, add a matching configs/zarr-<version>.txt, regenerate, and move the outgoing rows from the current table into the “Legacy results” table by hand (their flamegraphs are already committed).
Older versions and S3 are frozen in the “Legacy results” table and are not part of the default build. They can still be regenerated for reproducibility:
make legacy-profiles
export AWS_DEFAULT_REGION=...
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
make s3-profiles S3_PREFIX=s3://your-bucket/mem-array
make
This generates a temporal flamegraph for each current profile and then regenerates the current-results tables in this README.
To regenerate the tables without rebuilding flamegraphs:
make table
Flamegraph links point at the GitHub Pages copy (https://tomwhite.github.io/memray-array/), because github.com serves committed .html files as source rather than rendering them. To check flamegraphs that haven’t been pushed yet, either open flamegraphs/*.html directly in a browser, or generate the tables with relative links:
make table FLAMEGRAPH_BASE=flamegraphs
(Don’t commit the relative-link version — it renders as unclickable source on github.com.)