Serverless SQL on Zarr with map-lambda
Map-lambda isn't just good for rechunking - it can do SQL too! Two geospatial SQL queries over 3TB of real ERA5 data in S3 each ran from cold in a couple of minutes and cost a couple of dollars on AWS.
Map-lambda isn't just good for rechunking - it can do SQL too! Two geospatial SQL queries over 3TB of real ERA5 data in S3 each ran from cold in a couple of minutes and cost a couple of dollars on AWS.
Map-lambda is a new primitive operation for building data-intensive processing tools, like rechunking or resampling, that have a Python interface and a Rust backend running on AWS Lambda. A rechunk benchmark for 1.4TB of ERA5-shaped data showed a 3.7x performance increase over a similar benchmark from last year using Cubed on Lithops.