map-lambda¶
Fan out a Rust function to a thousand AWS Lambdas from Python in about a second.
The framework imposes very little overhead for latency:
1000 workers launch, run and return outputs in ~1s cold and ~0.5s warm (for a no-op worker)
While also being ideal for data-intensive applications:
a four stage rechunk of the original ERA5 dataset (1 variable) 1.4TB takes ~140s and costs ~$2.45; 3.7x faster than Cubed (from rechunk-lambda-benchmarks)
Note: This library is alpha-quality. The API and its behaviour may change in the future.
How it works¶
There are two levels of lambdas: a small number of dispatcher lambdas fan out to a large number of worker lambdas that run a pre-deployed compute function. The map inputs are split into partitions, and each partition is handled by a dispatcher.
Having dispatchers run in AWS removes the need for the client to manage a large number of network connections to the workers. The client still gets progress updates since they are sent periodically by the dispatchers over a small number of persistent HTTP connections.
There is no persistent state for coordination - everything flows through the lambda functions. As a result the system can scale to a thousand workers in ~1s.
Design¶
- AWS only. The implementation is deliberately targetted only at AWS Lambda and not other serverless function platforms, which avoids lowest-common denominator effects, and makes it easy to build on existing projects like Cargo Lambda. Knowledge of the limits in AWS Lambda (like number of concurrent lambdas, scaling rate, etc) can be built directly into the implementation.
- Rust on the backend, Python on the frontend. Taking advantage of Rust's high-performance and small binary size for AWS Lambdas means that cold start time is low (tens of milliseconds), while offering a Python API to make it easy to integrate in data processing applications. The client is Rust too; the Python API is a thin PyO3 binding over it.
- Execution not storage. Most map-lambda applications will process data on S3, however this is not a part of the library so it is up to applications to make sure the lambda is deployed in the same AWS region, manage S3 request rates, etc.