Skip to content

Usage

The map_lambda Python module provides a small API for running map functions. The following shows a simple example map for squaring integers.

import map_lambda

async for index, output in map_lambda.map_unordered("square", [1, 2, 3], input_key="input"):
    print(index, output)

square is the name of an AWS Lambda function, implemented in Rust using Cargo Lambda, which receives a request struct with a field containing the input value, and returns a response struct with the output:

use lambda_runtime::{Error, LambdaEvent};
use serde::{Deserialize, Serialize};

#[derive(Deserialize)]
pub(crate) struct Request {
    /// The input value being processed.
    input: i64,
}

#[derive(Serialize)]
pub(crate) struct Response {
    /// The output value of the function.
    output: i64,
}

pub(crate) async fn function_handler(event: LambdaEvent<Request>) -> Result<Response, Error> {
    let input = event.payload.input;
    let output = input * input;
    Ok(Response { output })
}

The worker lambda must be idempotent since inputs may be rerun in some circumstances (see "Failures and retries" below).

Extra arguments may be passed to each worker invocation via the constant_args Python argument. These are sent to the Rust Request, which must have the corresponding structure.

Inputs and outputs are sent as JSON, so they must be JSON-serialisable (e.g. Python pickle is not used). Inputs and outputs are used to coordinate a computation, so they should be small (a few KB) - large data values may be processed by providing a reference to them; they should not be passed as inputs or outputs.

map_unordered yields a pair per output: the position in inputs of the invocation that produced it, and a dict of the fields of the Rust Response object. Outputs arrive as the workers finish them, in no particular order, so index is what ties an output to its input. For the example above the output is some permutation of:

0 {'output': 1}
1 {'output': 4}
2 {'output': 9}

There is also a for_each function, which takes exactly the same arguments as map_unordered but doesn't return any values. The return type of the Rust function_handler in this case should be Result<(), Error> to indicate an empty response.

Functions deployed for a project have its name as a prefix, so two projects can each have a worker called copy without clashing. If memory is specified then it forms part of the function name too, so a call to map_unordered("square", ..., project="math", memory=1024) calls the lambda function math-square-1024.

Map-lambda provides diagnostics in the form of progress updates as the map_unordered (or for_each) call runs, dispatcher and worker invocation stats at the end of the run, and way to visualise the invocations over time. See the API doc and/or examples for details.

Failures and retries

A worker invocation is retried only if the request to Lambda fails, rather than the worker itself - for example if it is throttled, Lambda returns a service error, or the connection drops. After a dropped connection or a service error the worker may still have run, so retrying is permitted only because the function is idempotent. An error that another attempt can't change, like a missing permission or function, is not retried, so the call fails straight away.

The number of retries is set by the retries argument to the call, and defaults to 2. If the worker returns an Error it is not retried; it is assumed that the worker function will retry I/O errors itself.

The call fails with a RuntimeError in any of the following cases: - if the inputs cannot be dispatched, - if any invocation fails (a worker error, or a request to Lambda that still fails after retrying), - if there is a stream error, - if the stream does not end with a summary frame, - or if the dispatcher fails.

Once an invocation has failed, its dispatcher starts no more. Those already running finish, and other dispatchers run the rest of their partitions.

Out of scope

What is not guaranteed by the framework.

  • Dispatcher retries. There is no mechanism for retrying dispatchers. If a dispatcher fails then the call will fail and will need to be re-run by the client.
  • Cancellation. There is no cancellation mechanism, so once lambdas have started they will continue running in AWS. This can happen if the client fails or any of the connections to the dispatchers fail. A dispatcher does not find out that the client has gone, so stopping a call early still runs, and bills for, the rest of each partition.
  • Payload validation. It is up to the user to ensure that the deployed lambda function matches the Python frontend. A mismatch will be treated as a deserialisation error in the worker for each invocation.