Storage OS API

Lithops provides a transparent way to interact with the storage backend.

The module lithops.storage.cloud_proxy mimics the os and the built-in function open to access Cloud Storage as if it were a local file system.

By default, the configuration is loaded from the lithops config file, so there is no need to provide any parameter to use cloud_proxy:

from lithops.storage.cloud_proxy import os, open

with open('dir/file.txt', 'r') as f:
    content = f.read()

Alternatively, you can instantiate a CloudFileProxy with different lithops configuration through a dictionary. In this case, it will load the storage backend set in the storage key of the lithops section:

from lithops.storage.cloud_proxy import CloudStorage, CloudFileProxy

config = {'lithops' : {'storage_config' : 'ibm_cos'},
          'ibm_cos': {'region': 'REGION', 'api_key': 'API_KEY'}}

cloud_storage = CloudStorage(config)
cloud_file_proxy = CloudFileProxy(cloud_storage)

with cloud_file_proxy.open('dir/file.txt', 'r') as f:
    content = f.read()

Cloud Proxy Storage API

open()

Similar to Python’s built-in function open().

Manipulate an object stored in Cloud Object Storage.

Parameter

Description

file

File path. Must be an absolute path.

mode

Specify the mode in which the file is opened ('r' for read and 'w' for write text, 'b' for binary).

with open('bar/foo.txt', 'w') as f:
    f.write('Hello world!')

os

Similar to Python’s os library, except only file-related functionalities are implemented.

os.listdir

List all objects located in a directory.

Parameter

Description

Default

path

File path. Must be an absolute path.

suffix_dirs

Append a slash to directories listes

False

with open('bar/foo.txt', 'w') as f:
    f.write('Hello world!')

files = os.listdir('/bar')
print(files)  # Prints ['foo.txt']

os.walk

List recursively all files and directories in a root path.

Parameter

Description

Default

path

Root path. Must be an absolute path.

topdown

List directory tree top-down instead of bottom-up

True

files = ['/bar/foo.txt', '/bar/image.jpg', '/bar/subdir/data.csv']
for file in files:
    with open(file, 'w') as f:
        f.write('Hello world!')

for root, dirs, files in os.walk('/'):
    print(root, dirs, files)  # Prints '/' ['bar'] [], '/bar' ['subdir'] ['foo.txt', 'image.jpg'], '/bar/subdir' [] ['data.csv']

os.remove

Delete a file. If the directory where the file is located is empty after the file is deleted, this directory it is also removed.

Parameter

Description

Default

path

File path. Must be an absolute path.

with open('bar/foo.txt', 'w') as f:
    f.write('Hello world!')

os.remove('/bar/foo.txt')
files = os.listdir('/')
print(files)  # Prints []

os.path

Similar to Python’s os.path, except only file-realted functionalities are implemented.

os.path.isfile

Return True if a path is a file.

Parameter

Description

Default

path

File path. Must be an absolute path.

with open('bar/foo.txt', 'w') as f:
    f.write('Hello world!')

print(os.path.isfile('/bar/foo.txt'))  # Prints ``True``
print(os.path.isfile('/bar'))  # Prints ``False``

os.path.isdir

Return True if a path is a directory.

Parameter

Description

Default

path

Directory path. Must be an absolute path.

with open('bar/foo.txt', 'w') as f:
    f.write('Hello world!')

print(os.path.isdir('/bar/foo.txt'))  # Prints False
print(os.path.isdir('/bar'))  # Prints True

os.path.exists

Retrun True if a path corresponds to an existing file or directory in Cloud Object Storage.

Parameter

Description

Default

path

Target path. Must be an absolute path.

with open('bar/foo.txt', 'w') as f:
    f.write('Hello world!')

print(os.path.exists('/bar/foo.txt'))  # Prints True
print(os.path.exists('/baz/foo.txt'))  # Prints False