Storage API Details#

Lithops allows to create a Storage instance and abstract away the backend implementation details. The standard way to get a Storage object set up is to import the lithops Storage class and create an instance.

By default, the configuration is loaded from the lithops config file, so there is no need to provide any parameter to create a Storage instance:

from lithops import Storage

storage = Storage()

Alternatively, you can pass the lithops configuration through a dictionary. In this case, it will load the storage backend set in the storage key of the lithops section:

from lithops import Storage

config = {'lithops' : {'storage_config' : 'ibm_cos'},
          'ibm_cos': {'region': 'REGION', 'api_key': 'API_KEY'}}

storage = Storage(config=config)

In case you have multiple storage set in your configuration, you can force the storage backend by using the backend parameter:

from lithops import Storage

storage = Storage(backend='redis') # this will create a redis Storage instance

or:

from lithops import Storage

config = {'lithops' : {'storage_config' : 'ibm_cos'},
          'ibm_cos': {'region': 'REGION', 'api_key': 'API_KEY'},
          'redis': {'host': 'HOST', 'port':'PORT'}}


storage = Storage(config=config)  # this will create an ibm_cos Storage instance
storage = Storage(config=config, backend='redis')  # this will create a redis Storage instance

Storage API Reference#

class lithops.storage.storage.Storage(config=None, backend=None, storage_config=None)#

Bases: object

An Storage object is used by partitioner and other components to access underlying storage backend without exposing the the implementation details.

create_bucket(bucket)#

Creates a bucket if not exists.

Parameters:

bucket (str) – Name of the bucket

delete_cloudobject(cloudobject)#

Delete a CloudObject from storage.

Parameters:

cloudobject (CloudObject) – CloudObject instance

delete_cloudobjects(cloudobjects)#

Delete multiple CloudObjects from storage.

Parameters:

cloudobjects (List[CloudObject]) – List of CloudObject instances

delete_object(bucket, key)#

Removes objects from the storage backend.

Parameters:

  • bucket (str) – Name of the bucket

  • key (str) – Key of the object

delete_objects(bucket, key_list)#

This operation enables you to delete multiple objects from a bucket using a single HTTP request. If you know the object keys that you want to delete, then this operation provides a suitable alternative to sending individual delete requests, reducing per-request overhead.

Parameters:

  • bucket (str) – Name of the bucket

  • key_list (List[str]) – List of object keys

download_file(bucket, key, file_name=None, extra_args={}, config=None)#

Download a file from the storage backend. (Multipart download)

Parameters:

  • bucket (str) – Name of the bucket

  • key (str) – Key of the object

  • file_name (str | None) – Name of the file to save the object data

  • extra_args (Dict | None) – Extra get arguments to be passed to the underlying backend implementation (dict).

  • config (Any | None) – The transfer configuration to be used when performing the transfer (boto3.s3.transfer.TransferConfig).

Returns:

Object, as a binary array or as a file-like stream if parameter stream is enabled

Return type:

str | bytes | TextIO | BinaryIO

get_client()#

Retrieves the underlying storage client.

Returns:

Storage backend client

Return type:

object

get_cloudobject(cloudobject, stream=False)#

Get a CloudObject’s content from storage.

Parameters:

  • cloudobject (CloudObject) – CloudObject instance

  • stream (bool | None) – Get the object data or a file-like object

Returns:

Cloud object content

Return type:

str | bytes | TextIO | BinaryIO

get_object(bucket, key, stream=False, extra_get_args={})#

Retrieves objects from the storage backend.

Parameters:

  • bucket (str) – Name of the bucket

  • key (str) – Key of the object

  • stream (bool | None) – Get the object data or a file-like object

  • extra_get_args (Dict | None) – Extra get arguments to be passed to the underlying backend implementation (dict). For example, to specify the byte-range to read: extra_get_args={'Range': 'bytes=0-100'}.

Returns:

Object, as a binary array or as a file-like stream if parameter stream is enabled

Return type:

str | bytes | TextIO | BinaryIO

get_storage_config()#

Retrieves the configuration of this storage handler.

Returns:

Storage configuration

Return type:

Dict

head_bucket(bucket)#

This operation is useful to determine if a bucket exists and you have permission to access it. The operation returns a 200 OK if the bucket exists and you have permission to access it. Otherwise, the operation might return responses such as 404 Not Found and 403 Forbidden.

Parameters:

bucket (str) – Name of the bucket

Returns:

Request response

Return type:

Dict

head_object(bucket, key)#

The HEAD operation retrieves metadata from an object without returning the object itself. This operation is useful if you’re only interested in an object’s metadata.

Parameters:

  • bucket (str) – Name of the bucket

  • key (str) – Key of the object

Returns:

Object metadata

Return type:

Dict

list_keys(bucket, prefix=None)#

Similar to list_objects(), it returns all of the object keys in a bucket. For each object, the list contains only the names of the objects (keys).

Parameters:

  • bucket – Name of the bucket

  • prefix – Key prefix for filtering

Returns:

List of object keys

Return type:

List[str]

list_objects(bucket, prefix=None, match_pattern=None)#

Returns all of the object keys in a bucket. For each object, the list contains the name of the object (key) and the size.

Parameters:

  • bucket (str) – Name of the bucket

  • prefix (str | None) – Key prefix for filtering

  • match_pattern (str | None) –

Returns:

List of tuples containing the object key and size in bytes

Return type:

List[Tuple[str, int]]

put_cloudobject(body, bucket=None, key=None)#

Put a CloudObject into storage.

Parameters:

  • body (str | bytes | TextIO | BinaryIO) – Data content, can be a string or byte array or a text/bytes file-like object

  • bucket (str | None) – Destination bucket

  • key (str | None) – Destination key

Returns:

CloudObject instance

Return type:

CloudObject

put_object(bucket, key, body)#

Adds an object to a bucket of the storage backend.

Parameters:

  • bucket (str) – Name of the bucket

  • key (str) – Key of the object

  • body (str | bytes | TextIO | BinaryIO) – Object data

upload_file(file_name, bucket, key=None, extra_args={}, config=None)#

Upload a file to a bucket of the storage backend. (Multipart upload)

Parameters:

  • file_name (str) – Name of the file to upload

  • bucket (str) – Name of the bucket

  • key (str | None) – Key of the object

  • extra_args (Dict | None) – Extra get arguments to be passed to the underlying backend implementation (dict).

  • config (Any | None) – The transfer configuration to be used when performing the transfer (boto3.s3.transfer.TransferConfig).

Return type:

str | bytes | TextIO | BinaryIO