Storage API Deatils
Contents
Storage API Deatils#
Lithops allows to create a Storage instance and abstract away the backend implementation details.
The standard way to get a Storage object set up is to import the lithops Storage
class and create an instance.
By default, the configuration is loaded from the lithops config file, so there is no need to provide any parameter to create a Storage instance:
from lithops import Storage
storage = Storage()
Alternatively, you can pass the lithops configuration through a dictionary. In this case, it will load the storage
backend set in the storage
key of the lithops
section:
from lithops import Storage
config = {'lithops' : {'storage_config' : 'ibm_cos'},
'ibm_cos': {'region': 'REGION', 'api_key': 'API_KEY'}}
storage = Storage(config=config)
In case you have multiple storage set in your configuration, you can force the storage backend by
using the backend
parameter:
from lithops import Storage
storage = Storage(backend='redis') # this will create a redis Storage instance
or:
from lithops import Storage
config = {'lithops' : {'storage_config' : 'ibm_cos'},
'ibm_cos': {'region': 'REGION', 'api_key': 'API_KEY'},
'redis': {'host': 'HOST', 'port':'PORT'}}
storage = Storage(config=config) # this will create an ibm_cos Storage instance
storage = Storage(config=config, backend='redis') # this will create a redis Storage instance
Storage API Reference#
- class lithops.storage.storage.Storage(config=None, backend=None, storage_config=None)#
Bases:
object
An Storage object is used by partitioner and other components to access underlying storage backend without exposing the the implementation details.
- delete_cloudobject(cloudobject)#
Delete a CloudObject from storage.
- Parameters
cloudobject (lithops.storage.utils.CloudObject) – CloudObject instance
- delete_cloudobjects(cloudobjects)#
Delete multiple CloudObjects from storage.
- Parameters
cloudobjects (List[lithops.storage.utils.CloudObject]) – List of CloudObject instances
- delete_object(bucket, key)#
Removes objects from the storage backend.
- Parameters
bucket (str) – Name of the bucket
key (str) – Key of the object
- delete_objects(bucket, key_list)#
This operation enables you to delete multiple objects from a bucket using a single HTTP request. If you know the object keys that you want to delete, then this operation provides a suitable alternative to sending individual delete requests, reducing per-request overhead.
- Parameters
bucket (str) – Name of the bucket
key_list (List[str]) – List of object keys
- download_file(bucket, key, file_name=None, extra_args={})#
Download a file from the storage backend. (Multipart download)
- Parameters
bucket (str) – Name of the bucket
key (str) – Key of the object
file_name (Optional[str]) – Name of the file to save the object data
extra_args (Optional[Dict]) – Extra get arguments to be passed to the underlying backend implementation (dict).
- Returns
Object, as a binary array or as a file-like stream if parameter stream is enabled
- Return type
Union[str, bytes, TextIO, BinaryIO]
- get_client()#
Retrieves the underlying storage client.
- Returns
Storage backend client
- Return type
object
- get_cloudobject(cloudobject, stream=False)#
Get a CloudObject’s content from storage.
- Parameters
cloudobject (lithops.storage.utils.CloudObject) – CloudObject instance
stream (Optional[bool]) – Get the object data or a file-like object
- Returns
Cloud object content
- Return type
Union[str, bytes, TextIO, BinaryIO]
- get_object(bucket, key, stream=False, extra_get_args={})#
Retrieves objects from the storage backend.
- Parameters
bucket (str) – Name of the bucket
key (str) – Key of the object
stream (Optional[bool]) – Get the object data or a file-like object
extra_get_args (Optional[Dict]) – Extra get arguments to be passed to the underlying backend implementation (dict). For example, to specify the byte-range to read:
extra_get_args={'Range': 'bytes=0-100'}
.
- Returns
Object, as a binary array or as a file-like stream if parameter stream is enabled
- Return type
Union[str, bytes, TextIO, BinaryIO]
- get_storage_config()#
Retrieves the configuration of this storage handler.
- Returns
Storage configuration
- Return type
Dict
- head_bucket(bucket)#
This operation is useful to determine if a bucket exists and you have permission to access it. The operation returns a 200 OK if the bucket exists and you have permission to access it. Otherwise, the operation might return responses such as 404 Not Found and 403 Forbidden.
- Parameters
bucket (str) – Name of the bucket
- Returns
Request response
- Return type
Dict
- head_object(bucket, key)#
The HEAD operation retrieves metadata from an object without returning the object itself. This operation is useful if you’re only interested in an object’s metadata.
- Parameters
bucket (str) – Name of the bucket
key (str) – Key of the object
- Returns
Object metadata
- Return type
Dict
- list_keys(bucket, prefix=None)#
Similar to list_objects(), it returns all of the object keys in a bucket. For each object, the list contains only the names of the objects (keys).
- Parameters
bucket – Name of the bucket
prefix – Key prefix for filtering
- Returns
List of object keys
- Return type
List[str]
- list_objects(bucket, prefix=None, match_pattern=None)#
Returns all of the object keys in a bucket. For each object, the list contains the name of the object (key) and the size.
- Parameters
bucket (str) – Name of the bucket
prefix (Optional[str]) – Key prefix for filtering
match_pattern (Optional[str]) –
- Returns
List of tuples containing the object key and size in bytes
- Return type
List[Tuple[str, int]]
- put_cloudobject(body, bucket=None, key=None)#
Put a CloudObject into storage.
- Parameters
body (Union[str, bytes, TextIO, BinaryIO]) – Data content, can be a string or byte array or a text/bytes file-like object
bucket (Optional[str]) – Destination bucket
key (Optional[str]) – Destination key
- Returns
CloudObject instance
- Return type
lithops.storage.utils.CloudObject
- put_object(bucket, key, body)#
Adds an object to a bucket of the storage backend.
- Parameters
bucket (str) – Name of the bucket
key (str) – Key of the object
body (Union[str, bytes, TextIO, BinaryIO]) – Object data
- upload_file(file_name, bucket, key=None, extra_args={})#
Upload a file to a bucket of the storage backend. (Multipart upload)
- Parameters
file_name (str) – Name of the file to upload
bucket (str) – Name of the bucket
key (Optional[str]) – Key of the object
extra_args (Optional[Dict]) – Extra get arguments to be passed to the underlying backend implementation (dict).
- Return type
Union[str, bytes, TextIO, BinaryIO]