Functions and Parameters#

This document describes how to invoke functions based on the iterdata variable. In this sense, Lithops allows to send either args or kwargs in the function invocation. Take into account that there are some reserved parameter names that activate internal logic, and they cannot be used as regular parameters.

Reserved parameters#

Reserved parameters are only accessible when using the Futures API.

id: To get the call id. For instance, if you spawn 10 activations of a function, you will get here a number from 0 to 9, for example: map.py
obj: This parameter is used to activate the internal logic that allows to process data objects stored in the object store or public URLs in a transparent way. See data processing documentation for more details and instructions on how to use this built-in data-processing logic.
storage: To get a ready-to use lithops.storage.Storage() instance. This allows you to access your storage backend defined in configuration from any function in an easy way, for example: storage_arg.py
rabbitmq: To get a ready-to use pika.BlockingConnection() instance (AMQP URL must be set in the configuration to make it working). This allows you to access the RabbitMQ service from any function in an easy way, for example: rabbitmq_arg.py

Parameters in the call_async() method#

You can send multiple parameters to a single call function writing them into a list. The parameters will be mapped in the order you wrote them. In the following example the x parameter will take the value 3 and the y parameter will take the value 6.

import lithops

args = (3, 6)

def my_function(x, y):
    return x + y

fexec = lithops.FunctionExecutor()
fexec.call_async(my_function, args)
print (fexec.get_result())

The parameters can also be sent into a dictionary. In this case you have to map them to the correct parameter of the function as in the next example.

import lithops

kwargs = {'x': 2, 'y': 8}

def my_function(x, y):
    return x + y

fexec = lithops.FunctionExecutor()
fexec.call_async(my_function, kwargs)
print (fexec.get_result())

If you want to send a list or a dict as a parameter of the function, you must enclose them with [] as in the next example.

import lithops

args = ([1, 2, 3, 4, 5],)

def sum_list(list_of_numbers):
    total = 0
    for num in list_of_numbers:
        total = total+num
    return total

fexec = lithops.FunctionExecutor()
fexec.call_async(sum_list, args)
print (fexec.get_result())

You can also send multiple parameters which include a list.

import lithops

args = ([1, 2, 3, 4, 5], 5)

def sum_list_mult(list_of_numbers, x):
    total = 0
    for num in list_of_numbers:
        total = total+num
    return total*x

fexec = lithops.FunctionExecutor()
fexec.call_async(sum_list_mult, args)
print (fexec.get_result())

Or alternatively using a dict.

import lithops

kwargs = {'list_of_numbers': [1, 2, 3, 4, 5], 'x': 3}

fexec = lithops.FunctionExecutor()
fexec.call_async(sum_list_mult, kwargs)
print (fexec.get_result())

To test all of the previous examples run the multiple_args_call_async.py.

Parameters in the map() and map_reduce() methods#

The ‘iterdata’ variable must be always a list []. In this case to send multiple parameters to the function, the parameters of each function must be enclosed within a tuple () as in the next example. The parameters will be mapped in the order you wrote them.

import lithops

args = [  # Init list of parameters for Lithops
        (1, 2),  # Args for function1
        (3, 4),  # Args for function2
        (5, 6),  # Args for function3
       ]  # End list of parameters for Lithops

def my_function(x, y):
    return x + y

fexec = lithops.FunctionExecutor()
fexec.map(my_function, args)
print (fexec.get_result())

The parameters can also be sent into a dictionary. In this case you have to map them to the correct parameter of the function as in the next example.

import lithops

kwargs = [  # Init list of parameters for Lithops
          {'x': 1, 'y': 2},  # Kwargs for function1
          {'x': 3, 'y': 4},  # Kwargs for function2
          {'x': 5, 'y': 6},  # Kwargs for function3
         ]  # End list of parameters for Lithops

def my_function(x, y):
    return x + y

fexec = lithops.FunctionExecutor()
fexec.map(my_function, kwargs)
print (fexec.get_result())

If you want to send a list, a tuple or a dict as a parameter of the function, you must enclose them with () as in the next example.

import lithops

args = [  # Init list of parameters for Lithops
         ([1, 2],),  # Args for function1
         ([3, 4],),  # Args for function2
         ([5, 6],),  # Args for function3
       ]  # End list of parameters for Lithops

def sum_list(list_of_numbers):
    total = 0
    for num in list_of_numbers:
        total = total+num
    return total

fexec = lithops.FunctionExecutor()
fexec.map(sum_list, args)
print (fexec.get_result())

You can also send multiple parameters which include a list.

import lithops

args = [  # Init list of parameters for Lithops
        ([1, 2, 3, 4, 5], 2),  # Args for function1
        ([6, 7, 8, 9, 10], 3),  # Args for function2
        ([11, 12, 13, 14, 15], 4),  # Args for function3
       ]  # End list of parameters for Lithops

def sum_list_mult(list_of_numbers, x):
    total = 0
    for num in list_of_numbers:
        total = total+num
    return total*x

fexec = lithops.FunctionExecutor()
fexec.map(sum_list_mult, args)
print (fexec.get_result())

Or alternatively using a dict.

import lithops

kwargs = [  # Init list of parameters for Lithops
           {'list_of_numbers': [1, 2, 3, 4, 5], 'x': 2},  # Kwargs for function1
           {'list_of_numbers': [6, 7, 8, 9, 10], 'x': 3},  # Kwargs for function2
           {'list_of_numbers': [11, 12, 13, 14, 15], 'x': 4},  # Kwargs for function3
         ]  # End list of parameters for Lithops

fexec = lithops.FunctionExecutor()
fexec.map(sum_list_mult, kwargs)
print(fexec.get_result())

Common parameters across functions invocations#

Sometimes, functions have common parameters for all the invocations. In this case you have two options to proceed:

Setting variables in the global scope: You can define the desired variables in the global scope before defining the function. All of these variables can be catched within the function, for example:
```
import lithops

y = 10

def sum_x_y(x):
    return x+y

iterdata = [0, 1, 2]
fexec = lithops.FunctionExecutor()
fexec.map(sum_x_y, iterdata)
print(fexec.get_result())
```

Using extra_args parameter in the map() or map_reduce() calls. extra_args must be a *set or a dict, depending on whether iteradata contains args or kwargs.

If iterdata is a list of individual values or a list of sets:

import lithops

def sum_x_y(x, y):
    return x+y

args = [0, 1, 2]
fexec = lithops.FunctionExecutor()
fexec.map(sum_x_y, args, extra_args=(10,))
print(fexec.get_result())

The previous example is equivalent to the next:

import lithops

def sum_x_y(x, y):
    return x+y

args = [  # Init list of parameters for Lithops
        (0, 10),  # Args for function1
        (1, 10),  # Args for function2
        (2, 10),  # Args for function3
       ]  # End list of parameters for Lithops
fexec = lithops.FunctionExecutor()
fexec.map(sum_x_y, args)
print(fexec.get_result())

If iterdata is a list of dicts:

import lithops

kwargs = [  # Init list of parameters for Lithops
          {'x': 1},  # Kwargs for function1
          {'x': 3},  # Kwargs for function2
          {'x': 5},  # Kwargs for function3
         ]  # End list of parameters for Lithops

def my_function(x, y):
    return x + y

fexec = lithops.FunctionExecutor()
fexec.map(my_function, kwargs, extra_args={'y': 3})
print(fexec.get_result())

The previous example is equivalent to the next:

import lithops

kwargs = [  # Init list of parameters for Lithops
          {'x': 1, 'y': 3},  # Kwargs for function1
          {'x': 3, 'y': 3},  # Kwargs for function2
          {'x': 5, 'y': 3},  # Kwargs for function3
         ]  # End list of parameters for Lithops

def my_function(x, y):
    return x + y

fexec = lithops.FunctionExecutor()
fexec.map(my_function, kwargs)
print(fexec.get_result())

To test all of the previous examples run the multiple_args_map.py.

Functions and Parameters

Contents

Functions and Parameters#

Reserved parameters#

Parameters in the call_async() method#

Parameters in the map() and map_reduce() methods#

Common parameters across functions invocations#