Storage API Deatils#

Lithops allows to create a Storage instance and abstract away the backend implementation details. The standard way to get a Storage object set up is to import the lithops Storage class and create an instance.

By default, the configuration is loaded from the lithops config file, so there is no need to provide any parameter to create a Storage instance:

from lithops import Storage

storage = Storage()

Alternatively, you can pass the lithops configuration through a dictionary. In this case, it will load the storage backend set in the storage key of the lithops section:

from lithops import Storage

config = {'lithops' : {'storage_config' : 'ibm_cos'},
          'ibm_cos': {'region': 'REGION', 'api_key': 'API_KEY'}}

storage = Storage(config=config)

In case you have multiple storage set in your configuration, you can force the storage backend by using the backend parameter:

from lithops import Storage

storage = Storage(backend='redis') # this will create a redis Storage instance

or:

from lithops import Storage

config = {'lithops' : {'storage_config' : 'ibm_cos'},
          'ibm_cos': {'region': 'REGION', 'api_key': 'API_KEY'},
          'redis': {'host': 'HOST', 'port':'PORT'}}


storage = Storage(config=config)  # this will create an ibm_cos Storage instance
storage = Storage(config=config, backend='redis')  # this will create a redis Storage instance

Storage API Reference#

class lithops.storage.storage.Storage(config=None, backend=None, storage_config=None)#

Bases: object

An Storage object is used by partitioner and other components to access underlying storage backend without exposing the the implementation details.

delete_cloudobject(cloudobject)#

Delete a CloudObject from storage.

Parameters

cloudobject (lithops.storage.utils.CloudObject) – CloudObject instance

delete_cloudobjects(cloudobjects)#

Delete multiple CloudObjects from storage.

Parameters

cloudobjects (List[lithops.storage.utils.CloudObject]) – List of CloudObject instances

delete_object(bucket, key)#

Removes objects from the storage backend.

Parameters

  • bucket (str) – Name of the bucket

  • key (str) – Key of the object

delete_objects(bucket, key_list)#

This operation enables you to delete multiple objects from a bucket using a single HTTP request. If you know the object keys that you want to delete, then this operation provides a suitable alternative to sending individual delete requests, reducing per-request overhead.

Parameters

  • bucket (str) – Name of the bucket

  • key_list (List[str]) – List of object keys

download_file(bucket, key, file_name=None, extra_args={})#

Download a file from the storage backend. (Multipart download)

Parameters

  • bucket (str) – Name of the bucket

  • key (str) – Key of the object

  • file_name (Optional[str]) – Name of the file to save the object data

  • extra_args (Optional[Dict]) – Extra get arguments to be passed to the underlying backend implementation (dict).

Returns

Object, as a binary array or as a file-like stream if parameter stream is enabled

Return type

Union[str, bytes, TextIO, BinaryIO]

get_client()#

Retrieves the underlying storage client.

Returns

Storage backend client

Return type

object

get_cloudobject(cloudobject, stream=False)#

Get a CloudObject’s content from storage.

Parameters

  • cloudobject (lithops.storage.utils.CloudObject) – CloudObject instance

  • stream (Optional[bool]) – Get the object data or a file-like object

Returns

Cloud object content

Return type

Union[str, bytes, TextIO, BinaryIO]

get_object(bucket, key, stream=False, extra_get_args={})#

Retrieves objects from the storage backend.

Parameters

  • bucket (str) – Name of the bucket

  • key (str) – Key of the object

  • stream (Optional[bool]) – Get the object data or a file-like object

  • extra_get_args (Optional[Dict]) – Extra get arguments to be passed to the underlying backend implementation (dict). For example, to specify the byte-range to read: extra_get_args={'Range': 'bytes=0-100'}.

Returns

Object, as a binary array or as a file-like stream if parameter stream is enabled

Return type

Union[str, bytes, TextIO, BinaryIO]

get_storage_config()#

Retrieves the configuration of this storage handler.

Returns

Storage configuration

Return type

Dict

head_bucket(bucket)#

This operation is useful to determine if a bucket exists and you have permission to access it. The operation returns a 200 OK if the bucket exists and you have permission to access it. Otherwise, the operation might return responses such as 404 Not Found and 403 Forbidden.

Parameters

bucket (str) – Name of the bucket

Returns

Request response

Return type

Dict

head_object(bucket, key)#

The HEAD operation retrieves metadata from an object without returning the object itself. This operation is useful if you’re only interested in an object’s metadata.

Parameters

  • bucket (str) – Name of the bucket

  • key (str) – Key of the object

Returns

Object metadata

Return type

Dict

list_keys(bucket, prefix=None)#

Similar to list_objects(), it returns all of the object keys in a bucket. For each object, the list contains only the names of the objects (keys).

Parameters

  • bucket – Name of the bucket

  • prefix – Key prefix for filtering

Returns

List of object keys

Return type

List[str]

list_objects(bucket, prefix=None, match_pattern=None)#

Returns all of the object keys in a bucket. For each object, the list contains the name of the object (key) and the size.

Parameters

  • bucket (str) – Name of the bucket

  • prefix (Optional[str]) – Key prefix for filtering

  • match_pattern (Optional[str]) –

Returns

List of tuples containing the object key and size in bytes

Return type

List[Tuple[str, int]]

put_cloudobject(body, bucket=None, key=None)#

Put a CloudObject into storage.

Parameters

  • body (Union[str, bytes, TextIO, BinaryIO]) – Data content, can be a string or byte array or a text/bytes file-like object

  • bucket (Optional[str]) – Destination bucket

  • key (Optional[str]) – Destination key

Returns

CloudObject instance

Return type

lithops.storage.utils.CloudObject

put_object(bucket, key, body)#

Adds an object to a bucket of the storage backend.

Parameters

  • bucket (str) – Name of the bucket

  • key (str) – Key of the object

  • body (Union[str, bytes, TextIO, BinaryIO]) – Object data

upload_file(file_name, bucket, key=None, extra_args={})#

Upload a file to a bucket of the storage backend. (Multipart upload)

Parameters

  • file_name (str) – Name of the file to upload

  • bucket (str) – Name of the bucket

  • key (Optional[str]) – Key of the object

  • extra_args (Optional[Dict]) – Extra get arguments to be passed to the underlying backend implementation (dict).

Return type

Union[str, bytes, TextIO, BinaryIO]