Lithops Core API

Lithops Core API#

The core abstraction in Lithops is the executor, which is responsible for orchestrating the execution of your functions across different environments.

To get started, you typically import lithops and create an executor instance to run your code. Lithops provides a flexible set of executors to suit different needs.

Primary Executors#

  • FunctionExecutor (lithops.FunctionExecutor()): The main, generic executor that automatically selects its execution mode based on the provided configuration. This lets you write your code once and run it seamlessly on localhost, serverless, or standalone backends without changing your code.

  • RetryingFunctionExecutor (lithops.RetryingFunctionExecutor()): A robust wrapper around FunctionExecutor that transparently handles retries on failed tasks. It supports all features of FunctionExecutor with added automatic retry logic, improving fault tolerance and reliability for unstable or transient-failure-prone environments.

Secondary Executors#

For more specialized use cases, Lithops also provides explicit executors for each execution mode:

  • LocalhostExecutor (lithops.LocalhostExecutor()): Runs jobs locally using multiple processes on your machine. Ideal for development, debugging, or small-scale workloads.

  • ServerlessExecutor (lithops.ServerlessExecutor()): Executes jobs on serverless compute platforms, managing scaling and deployment automatically. Best for massively parallel, ephemeral workloads.

  • StandaloneExecutor (lithops.StandaloneExecutor()): Runs jobs on standalone compute backends such as clusters or virtual machines. Suitable for long-running or resource-heavy tasks.

Configuration and Initialization#

By default, executors load configuration from the Lithops configuration file (e.g., ~/.lithops/config). You can also supply configuration parameters programmatically via a Python dictionary when creating an executor instance. Parameters passed explicitly override those in the config file, allowing for flexible customization on the fly.

This layered executor design lets Lithops provide a powerful, unified API for parallel function execution — from local development to multi-cloud production deployments, with fault tolerance and retries built in.

Core API Reference#

class lithops.executors.FunctionExecutor(mode=None, config=None, config_file=None, backend=None, storage=None, monitoring=None, log_level=False, **kwargs)#

Bases: object

Base executor that contains the common logic for the Localhost, Serverless and Standalone executors.

Parameters:

  • mode (str | None) – Execution mode. One of: localhost, serverless or standalone

  • config (Dict[str, Any] | None) – Settings passed in here will override those in lithops_config

  • config_file (str | None) – Path to the lithops config file

  • backend (str | None) – Compute backend to run the functions

  • storage (str | None) – Storage backend to store Lithops data

  • monitoring (str | None) – Monitoring backend name. Built-in: storage, rabbitmq, redis, aws_sqs, gcp_pubsub, azure_queue

  • log_level (str | bool | None) – Log level printing (INFO, DEBUG, …). Set it to None to hide all logs. If this is param is set, all logging params in config are disabled

  • kwargs (Any) – Any parameter that can be set in the compute backend section of the config file, can be set here

call_async(func, data, extra_env=None, runtime_memory=None, timeout=None, include_modules=[], exclude_modules=[])#

For running one function execution asynchronously.

Parameters:

  • func (Callable) – The function to map over the data.

  • data (List[Any] | Tuple[Any, ...] | Dict[str, Any]) – Input data. Arguments can be passed as a list or tuple, or as a dictionary for keyword arguments.

  • extra_env (Dict | None) – Additional env variables for function environment.

  • runtime_memory (int | None) – Memory to use to run the function.

  • timeout (int | None) – Time that the function has to complete its execution before raising a timeout.

  • include_modules (List | None) – Explicitly pickle these dependencies.

  • exclude_modules (List | None) – Explicitly keep these modules from pickled dependencies.

Returns:

Response future.

Return type:

ResponseFuture

clean(fs=None, cs=None, clean_cloudobjects=True, clean_fn=False, force=False, on_exit=False)#

Deletes all the temp files from storage. These files include the function, the data serialization and the function invocation results. It can also clean cloudobjects.

Parameters:

  • fs (ResponseFuture | List[ResponseFuture] | None) – List of futures to clean

  • cs (List[CloudObject] | None) – List of cloudobjects to clean

  • clean_cloudobjects (bool | None) – Delete all cloudobjects created with this executor

  • clean_fn (bool | None) – Delete cached functions in this executor

  • force (bool | None) – Clean all future objects even if they have not been completed

  • on_exit (bool | None) – do not print logs on exit

get_result(fs=None, throw_except=True, timeout=None, threadpool_size=64, wait_dur_sec=None, show_progressbar=True)#

For getting the results from all function activations

Parameters:

  • fs (ResponseFuture | FuturesList | List[ResponseFuture] | None) – Futures list. Default None

  • throw_except (bool | None) – Reraise exception if call raised. Default True.

  • timeout (int | None) – Timeout for waiting for results.

  • threadpool_size (int | None) – Number of threads to use. Default 64

  • wait_dur_sec (int | None) – Time interval between each check. Default 1 second

  • show_progressbar (bool | None) – whether or not to show the progress bar.

Returns:

The result of the future/s

job_summary(cloud_objects_n=0)#

Logs information of a job executed by the calling function executor. currently supports: code_engine, ibm_vpc and ibm_cf.

Parameters:

cloud_objects_n (int | None) – number of cloud object used in COS, declared by user.

map(map_function, map_iterdata, chunksize=None, extra_args=None, extra_env=None, runtime_memory=None, obj_chunk_size=None, obj_chunk_number=None, obj_newline='\n', timeout=None, include_modules=[], exclude_modules=[])#

Spawn multiple function activations based on the items of an input list.

Parameters:

  • map_function (Callable) – The function to map over the data

  • map_iterdata (List[List[Any] | Tuple[Any, ...] | Dict[str, Any]]) – An iterable of input data (e.g python list).

  • chunksize (int | None) – Split map_iterdata in chunks of this size. Lithops spawns 1 worker per resulting chunk

  • extra_args (List[Any] | Tuple[Any, ...] | Dict[str, Any] | None) – Additional arguments to pass to each map_function activation

  • extra_env (Dict[str, str] | None) – Additional environment variables for function environment

  • runtime_memory (int | None) – Memory (in MB) to use to run the functions

  • obj_chunk_size (int | None) – Used for data processing. Chunk size to split each object in bytes. Must be >= 1MiB. ‘None’ for processing the whole file in one function activation

  • obj_chunk_number (int | None) – Used for data processing. Number of chunks to split each object. ‘None’ for processing the whole file in one function activation. chunk_n has prevalence over chunk_size if both parameters are set

  • obj_newline (str | None) – new line character for keeping line integrity of partitions. ‘None’ for disabling line integrity logic and get partitions of the exact same size in the functions

  • timeout (int | None) – Max time per function activation (seconds)

  • include_modules (List[str] | None) – Explicitly pickle these dependencies. All required dependencies are pickled if default empty list. No one dependency is pickled if it is explicitly set to None

  • exclude_modules (List[str] | None) – Explicitly keep these modules from pickled dependencies. It is not taken into account if you set include_modules.

Returns:

A list with size len(map_iterdata) of futures for each job (Futures are also internally stored by Lithops).

Return type:

FuturesList

map_reduce(map_function, map_iterdata, reduce_function, chunksize=None, extra_args=None, extra_args_reduce=None, extra_env=None, map_runtime_memory=None, reduce_runtime_memory=None, timeout=None, obj_chunk_size=None, obj_chunk_number=None, obj_newline='\n', obj_reduce_by_key=False, spawn_reducer=20, include_modules=[], exclude_modules=[])#

Map the map_function over the data and apply the reduce_function across all futures.

Parameters:

  • map_function (Callable) – The function to map over the data

  • map_iterdata (List[List[Any] | Tuple[Any, ...] | Dict[str, Any]]) – An iterable of input data

  • reduce_function (Callable) – The function to reduce over the futures

  • chunksize (int | None) – Split map_iterdata in chunks of this size. Lithops spawns 1 worker per resulting chunk. Default 1

  • extra_args (List[Any] | Tuple[Any, ...] | Dict[str, Any] | None) – Additional arguments to pass to function activation. Default None

  • extra_args_reduce (List[Any] | Tuple[Any, ...] | Dict[str, Any] | None) – Additional arguments to pass to the reduce function activation. Default None

  • extra_env (Dict[str, str] | None) – Additional environment variables for action environment. Default None

  • map_runtime_memory (int | None) – Memory to use to run the map function. Default None (loaded from config)

  • reduce_runtime_memory (int | None) – Memory to use to run the reduce function. Default None (loaded from config)

  • timeout (int | None) – Time that the functions have to complete their execution before raising a timeout

  • obj_chunk_size (int | None) – the size of the data chunks to split each object. ‘None’ for processing the whole file in one function activation

  • obj_chunk_number (int | None) – Number of chunks to split each object. ‘None’ for processing the whole file in one function activation

  • obj_newline (str | None) – New line character for keeping line integrity of partitions. ‘None’ for disabling line integrity logic and get partitions of the exact same size in the functions

  • obj_reduce_by_key (bool | None) – Set one reducer per object after running the partitioner. By default there is one reducer for all the objects

  • spawn_reducer (int | None) – Percentage of done map functions before spawning the reduce function

  • include_modules (List[str] | None) – Explicitly pickle these dependencies.

  • exclude_modules (List[str] | None) – Explicitly keep these modules from pickled dependencies.

Returns:

A list with size len(map_iterdata) of futures.

Return type:

FuturesList

plot(fs=None, dst=None, figsize=(10, 6))#

Creates timeline and histogram of the current execution in dst.

Parameters:

  • fs (ResponseFuture | FuturesList | List[ResponseFuture] | None) – list of futures.

  • dst (str | None) – destination path to save .png plots.

  • figsize (tuple | None) – size of the plots, in inches.

wait(fs=None, throw_except=True, return_when=100, download_results=False, timeout=None, threadpool_size=64, wait_dur_sec=None, show_progressbar=True, clean_jobs=None)#

Wait for the Future instances (possibly created by different Executor instances) given by fs to complete. Returns a named 2-tuple of sets. The first set, named done, contains the futures that completed (finished or cancelled futures) before the wait completed. The second set, named not_done, contains the futures that did not complete (pending or running futures). timeout can be used to control the maximum number of seconds to wait before returning.

Parameters:

  • fs (ResponseFuture | FuturesList | List[ResponseFuture] | None) – Futures list. Default None

  • throw_except (bool | None) – Re-raise exception if call raised. Default True

  • return_when (Any | None) – Percentage of done futures

  • download_results (bool | None) – Download results. Default false (Only get statuses)

  • timeout (int | None) – Timeout of waiting for results

  • threadpool_size (int | None) – Number of threads to use. Default 64

  • wait_dur_sec (int | None) – Time interval between each check. Default 1 second

  • show_progressbar (bool | None) – whether or not to show the progress bar.

  • clean_jobs (bool | None) – Delete temporary job data when every future is done. Default is data_cleaner. get_result() passes False so it can log that results are in before it cleans.

Returns:

(fs_done, fs_notdone) where fs_done is a list of futures that have completed and fs_notdone is a list of futures that have not completed.

Return type:

Tuple[FuturesList, FuturesList]

class lithops.executors.LocalhostExecutor(config=None, config_file=None, storage=None, monitoring=None, log_level=False, **kwargs)#

Bases: FunctionExecutor

Initialize a LocalhostExecutor class.

Parameters:

  • config (Dict[str, Any] | None) – Settings passed in here will override those in config file.

  • config_file (str | None) – Path to the lithops config file

  • storage (str | None) – Name of the storage backend to use.

  • monitoring (str | None) – monitoring system.

  • log_level (str | bool | None) – log level to use during the execution.

  • kwargs (Any) – Any parameter that can be set in the compute backend section of the config file, can be set here

class lithops.executors.ServerlessExecutor(config=None, config_file=None, backend=None, storage=None, monitoring=None, log_level=False, **kwargs)#

Bases: _FixedModeExecutor

Initialize a ServerlessExecutor class.

Parameters:

  • config (Dict[str, Any] | None) – Settings passed in here will override those in config file

  • config_file (str | None) – Path to the lithops config file

  • backend (str | None) – Name of the serverless compute backend to use

  • storage (str | None) – Name of the storage backend to use

  • monitoring (str | None) – monitoring system

  • log_level (str | bool | None) – log level to use during the execution

  • kwargs (Any) – Any parameter that can be set in the compute backend section of the config file, can be set here

class lithops.executors.StandaloneExecutor(config=None, config_file=None, backend=None, storage=None, monitoring=None, log_level=False, **kwargs)#

Bases: _FixedModeExecutor

Initialize a StandaloneExecutor class.

Parameters:

  • config (Dict[str, Any] | None) – Settings passed in here will override those in config file

  • config_file (str | None) – Path to the lithops config file

  • backend (str | None) – Name of the standalone compute backend to use

  • storage (str | None) – Name of the storage backend to use

  • monitoring (str | None) – monitoring system

  • log_level (str | bool | None) – log level to use during the execution

  • kwargs (Any)

class lithops.retries.RetryingFunctionExecutor(executor)#

Bases: object

Wrapper around FunctionExecutor with automatic retries.

Same public interface as FunctionExecutor, plus a retries parameter on map() for per-invocation retry budget.

Parameters:

executor (FunctionExecutor) – FunctionExecutor (localhost, serverless, or standalone)

clean(fs=None, cs=None, clean_cloudobjects=True, clean_fn=False, force=False)#

Cleans up temporary files and objects related to this executor, including: - Function packages - Serialized input/output data - Cloud objects (if specified)

Parameters:

  • fs (ResponseFuture | List[ResponseFuture] | None) – List of futures to clean.

  • cs (List[CloudObject] | None) – List of cloudobjects to clean.

  • clean_cloudobjects (bool | None) – Whether to delete all cloudobjects created with this executor.

  • clean_fn (bool | None) – Whether to delete cached functions.

  • force (bool | None) – Force cleanup even for unfinished futures.

map(map_function, map_iterdata, chunksize=None, extra_args=None, extra_env=None, runtime_memory=None, obj_chunk_size=None, obj_chunk_number=None, obj_newline='\n', timeout=None, include_modules=[], exclude_modules=[], retries=None)#

Spawn multiple function activations with automatic retry on failure.

Parameters:

  • map_function (Callable) – The function to map over the data.

  • map_iterdata (List[List[Any] | Tuple[Any, ...] | Dict[str, Any]]) – An iterable of input data (e.g., Python list).

  • chunksize (int | None) – Split map_iterdata in chunks of this size. One worker per chunk.

  • extra_args (List[Any] | Tuple[Any, ...] | Dict[str, Any] | None) – Additional arguments to pass to each function.

  • extra_env (Dict[str, str] | None) – Additional environment variables for the function environment.

  • runtime_memory (int | None) – Memory (in MB) to allocate per function activation.

  • obj_chunk_size (int | None) – For file processing. Split each object into chunks of this size (in bytes).

  • obj_chunk_number (int | None) – For file processing. Number of chunks to split each object into.

  • obj_newline (str | None) – Newline character for line integrity in file partitioning.

  • timeout (int | None) – Max time per function activation (in seconds).

  • include_modules (List[str] | None) – Explicitly pickle these dependencies.

  • exclude_modules (List[str] | None) – Explicitly exclude these modules from pickling.

  • retries (int | None) – Number of retries for each function activation upon failure.

Returns:

A list of RetryingFuture objects, one for each function activation.

Return type:

List[RetryingFuture]

wait(fs, throw_except=True, return_when=100, download_results=False, timeout=None, threadpool_size=64, wait_dur_sec=1, show_progressbar=True)#

Wait for a set of futures to complete, retrying any that fail.

Parameters:

  • fs (List[RetryingFuture]) – List of RetryingFuture objects to wait on.

  • throw_except (bool | None) – Raise exceptions encountered during execution.

  • return_when (Any | None) – Completion policy. One of: ALWAYS, ANY_COMPLETED, or ALL_COMPLETED.

  • download_results (bool | None) – Whether to download results after completion.

  • timeout (int | None) – Maximum wait time (in seconds).

  • threadpool_size (int | None) – Number of threads used for polling.

  • wait_dur_sec (int | None) – Polling interval (in seconds).

  • show_progressbar (bool | None) – Show progress bar for the wait operation.

Returns:

A tuple (done, pending) of lists of RetryingFutures.

Return type:

Tuple[List[RetryingFuture], List[RetryingFuture]]

map() and map_reduce() return a FuturesList, which is a list of futures that can be mapped over again. A plain list or tuple of those futures is a chain too, including a slice or a comprehension. See Function chaining.

class lithops.utils.FuturesList(iterable=(), /)#

Bases: list

List of futures that can be mapped over again, so that jobs can be chained. Chaining replaces the contents with the futures of the new job, while alt_list keeps every future of the chain for wait() and get_result()

get_result(**kwargs)#

Returns the results of the last job of the chain. The intermediate ones are read by the workers, never by the client.

Parameters:

kwargs – Passed on to get_result()

Returns:

The results of the last job

map(map_function, sync=False, **kwargs)#

Chains a new map job that takes the results of this one as its input. The intermediate results are read by the workers of the new job and are never downloaded to the client.

Parameters:

  • map_function – The function to map over the results

  • sync – Wait for this job before invoking the new one. Left False, the new job is invoked right away and each of its workers blocks until its own input is ready, which is worker time you pay for. Waiting costs about the same wall-clock time and no idle workers, at the price of not overlapping the two invocations

  • kwargs – Passed on to map(). extra_args is not among them: a chained function is called with the result of the previous one and nothing else

Returns:

This list, now holding the futures of the new job

map_reduce(map_function, reduce_function, sync=False, **kwargs)#

Chains a new map-reduce job that takes the results of this one as the input of its map stage.

Parameters:

  • map_function – The function to map over the results

  • reduce_function – The function to reduce the map results with

  • sync – Wait for this job before invoking the new one. See map()

  • kwargs – Passed on to map_reduce()

Returns:

This list, now holding the futures of the new job

wait(**kwargs)#

Waits for every job of the chain, not only for the last one.

Parameters:

kwargs – Passed on to wait()

Returns:

(fs_done, fs_notdone)