IBM Cloud Object Storage#

Lithops with IBM COS as storage backend.

Installation#

Install IBM Cloud backend dependencies:

python3 -m pip install lithops[ibm]

Create an IBM Cloud Object Storage account.
Create a bucket in your desired region. Remember to update the corresponding Lithops config field with this bucket name.

Configuration#

Choose one authentication option below.

Option 1 (COS API Key):#

In the side navigation, click Service Credentials.
Click New credential + and provide the necessary information.
Click Add to generate service credential.
Click View credentials and copy the apikey value.

Edit your Lithops config file and add the following keys:

lithops:
    storage: ibm_cos
   
ibm_cos:
    region   : <REGION>
    api_key  : <API_KEY>
    storage_bucket: <BUCKET_NAME>

Option 2 (COS HMAC credentials):#

In the side navigation, click Service Credentials.
Click New credential +.
Click on advanced options and enable Include HMAC Credential button.
Click Add to generate service credential.
Click View credentials and copy the access_key_id and secret_access_key values.
When using HMAC credentials, you can omit providing a storage bucket, since Lithops will be able to create it automatically.

Edit your Lithops config file and add the following keys:

lithops:
    storage: ibm_cos
   
ibm_cos:
    region: <REGION>  
    access_key_id: <ACCESS_KEY_ID>
    secret_access_key: <SECRET_ACCESS_KEY>

Option 3 (IBM IAM API Key):#

If you don’t have an IAM API key created, navigate to the IBM IAM dashboard
Click Create an IBM Cloud API Key and provide the necessary information.
Copy the generated IAM API key (you can only see the key the first time you create it, so make sure to copy it).

Edit your Lithops config file and add the following keys:

lithops:
    storage: ibm_cos

ibm:
    iam_api_key: <IAM_API_KEY>

ibm_cos:
    region: <REGION>
    storage_bucket: <BUCKET_NAME>

Lithops COS Endpoint configuration#

Using region (recommended)#

The easiest approach is to let Lithops choose the right endpoint by itself. Configure Lithops with the region name of your storage_bucket:

    ibm_cos:
        region   : <REGION>

Valid region names include: us-east, us-south, eu-gb, eu-de, eu-es, ca-tor, br-sao, jp-tok, jp-osa, au-syd.

When region is set, Lithops automatically configures:

Public endpoint (client): https://s3.<region>.cloud-object-storage.appdomain.cloud
Worker endpoint (code_engine and ibm_vpc): https://s3.direct.<region>.cloud-object-storage.appdomain.cloud

You do not need to set endpoint or private_endpoint manually when using region with these backends.

Using endpoint paths#

As an alternative to using region, you can configure the public and worker endpoints explicitly:

Login to IBM Cloud and open up your dashboard. Then navigate to your instance of Object Storage.
In the side navigation, click Endpoints to find your COS endpoints. Copy the endpoint for the region where you created your bucket.

    ibm_cos:
        endpoint: https://s3.<region>.cloud-object-storage.appdomain.cloud
        private_endpoint: https://s3.direct.<region>.cloud-object-storage.appdomain.cloud

Use the direct endpoint (s3.direct) as private_endpoint when running Lithops with code_engine or ibm_vpc.

Summary of configuration keys for IBM Cloud:#

IBM IAM:#

Group	Key	Mandatory	Additional info
ibm	iam_api_key	no	IBM Cloud IAM API key to authenticate against IBM services. Obtain the key here
ibm	region	no	IBM Region. One of: `eu-gb`, `eu-de`, `eu-es`, `us-south`, `us-east`, `br-sao`, `ca-tor`, `jp-tok`, `jp-osa`, `au-syd`
ibm	resource_group_id	no	Resource group id from your IBM Cloud account. Get it from here