IBM Cloud Object Storage#

Lithops with IBM COS as storage backend.

Installation#

  1. Create an IBM Cloud Object Storage account.

  2. Crate a bucket in your desired region. Remember to update the corresponding Lithops config field with this bucket name.

Configuration#

  1. Create the credentials to access to your COS account (Choose one option):

Option 1 (COS API Key):#

  1. In the side navigation, click Service Credentials.

  2. Click New credential + and provide the necessary information.

  3. Click Add to generate service credential.

  4. Click View credentials and copy the apikey value.

  5. Edit your lithops config file and add the following keys:

    lithops:
        storage: ibm_cos
       
    ibm_cos:
        storage_bucket: <BUCKET_NAME>
        region   : <REGION>
        api_key  : <API_KEY>
    

Option 2 (COS HMAC credentials):#

  1. In the side navigation, click Service Credentials.

  2. Click New credential +.

  3. Click on advanced options and enable Include HMAC Credential button.

  4. Click Add to generate service credential.

  5. Click View credentials and copy the access_key_id and secret_access_key values.

  6. Edit your lithops config file and add the following keys:

    lithops:
        storage: ibm_cos
       
    ibm_cos:
        storage_bucket: <BUCKET_NAME>
        region : <REGION>  
        access_key  : <ACCESS_KEY_ID>
        secret_key  : <SECRET_KEY_ID>
    

Option 3 (IBM IAM API Key):#

  1. If you don’t have an IAM API key created, navigate to the IBM IAM dashboard

  2. Click Create an IBM Cloud API Key and provide the necessary information.

  3. Copy the generated IAM API key (You can only see the key the first time you create it, so make sure to copy it).

  4. Edit your lithops config file and add the following keys:

    lithops:
        storage_backend: ibm_cos
    
    ibm:
        iam_api_key: <IAM_API_KEY>
       
    ibm_cos:
        storage_bucket: <BUCKET_NAME>
        region : <REGION>
    

Lithops COS Endpoint configuration#

Using region#

The easiest apporach is to let Lithops to choose the right endpoint by itself. To enable this, just configure Lithops with the region name of your storage_bucket, as follows:

    ibm_cos:
        region   : <REGION>

Valid region names are: us-east, us-south, eu-gb, eu-de, etc..

Using endpoints path#

Alternative to using region, you can configure the public and private endpoints as follows:

  1. Login to IBM Cloud and open up your dashboard. Then navigate to your instance of Object Storage.

  2. In the side navigation, click Endpoints to find your COS endpoints. You must copy both public and private endpoints of the region where you created your bucket.

    ibm_cos:
        endpoint: https://s3.<region>.cloud-object-storage_config.appdomain.cloud
        private_endpoint: https://s3.private.<region>.cloud-object-storage_config.appdomain.cloud 

Summary of configuration keys for IBM Cloud:#

IBM IAM:#

Group

Key

Default

Mandatory

Additional info

ibm

iam_api_key

no

IBM Cloud IAM API key to authenticate against IBM COS and IBM Cloud Functions. Obtain the key here

ibm

iam_endpoint

https://iam.cloud.ibm.com

no

IAM Identity Services API global endpoint URL.

IBM Cloud Object Storage:#

Group

Key

Default

Mandatory

Additional info

ibm_cos

storage_bucket

yes

The name of a bucket that exists in you account. This will be used by Lithops for intermediate data. If set, this will overwrite the storage_bucket set in lithops section

ibm_cos

region

no

Region of your bucket. Mandatory if no endpoint. For example: us-east, us-south, eu-gb, eu-de, etc..

ibm_cos

endpoint

no

Endpoint to your COS account. Mandatory if no region. Make sure to use the full path with ‘https://’ as prefix.

ibm_cos

private_endpoint

no

Private endpoint to your COS account. Mandatory if no region. Make sure to use the full path with ‘https://’ or http:// as prefix.

ibm_cos

api_key

no

API Key to your COS account. Mandatory if no access_key and secret_key. Not needed if using IAM API Key

ibm_cos

access_key

no

HMAC Credentials. Mandatory if no api_key. Not needed if using IAM API Key

ibm_cos

secret_key

no

HMAC Credentials. Mandatory if no api_key. Not needed if using IAM API Key

ibm_cos

iam_api_key

no

IBM Cloud IAM API key to authenticate against IBM COS. Used in case separate authorization for storage and compute required