Google Compute Engine (GCE)

The GCP Compute Engine backend of Lithops can provide a serverless user experience on top of GCE where Lithops creates new Virtual Machines (VMs) dynamically at runtime and scales Lithops jobs against them (create and reuse modes). Alternatively Lithops can start and stop an existing VM instance (consume mode).

The backend key is gcp_compute_engine (matches the Lithops module name).

Choose an operating system image for the VM

Any VM needs an operating system image. By default Lithops uses Ubuntu 24.04 (projects/ubuntu-os-cloud/global/images/family/ubuntu-2404-lts-amd64). Lithops installs required dependencies on the VM on first use (this can take a few minutes).

For faster startups, build a pre-configured custom image (see runtime/gcp_compute_engine):

lithops image build -b gcp_compute_engine

This creates lithops-ubuntu-2404-lts-amd64-server in your project; Lithops uses it automatically when present.

To list available images:

lithops image list -b gcp_compute_engine

Use the Image ID column as source_image in your config when using a custom image name.

Installation

1. Install GCP backend dependencies:

python3 -m pip install lithops[gcp]

Create and reuse modes

In the create mode, Lithops automatically creates new worker VM instances at runtime, runs the job on them, and deletes the workers when the job completes (unless configured otherwise).

In the reuse mode, Lithops keeps the master and worker VMs stopped between jobs and starts them again when needed. It reuses running workers when possible and only creates new workers if necessary.

Configuration

1. Enable the Compute Engine API:

gcloud services enable compute.googleapis.com --project <PROJECT_ID>

2. Create a service account (or use an existing one) and grant these roles on the project:

gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member="serviceAccount:<SERVICE_ACCOUNT_EMAIL>" \
  --role="roles/compute.admin"

gcloud projects add-iam-policy-binding <PROJECT_ID> \
  --member="serviceAccount:<SERVICE_ACCOUNT_EMAIL>" \
  --role="roles/storage.objectAdmin"

3. Set the Lithops config file:

lithops:
  backend: gcp_compute_engine

gcp:
  credentials_path: <FULL_PATH_TO_SERVICE_ACCOUNT_JSON>

gcp_compute_engine:
  project_name: <GCP_PROJECT_ID>
  zone: <ZONE>
  exec_mode: reuse

Lithops attaches the service account from credentials_path to master and worker VMs so they can access GCS via the metadata service. You can set service_account: <EMAIL> explicitly if needed.

Summary of configuration keys for GCP

Group	Key	Default	Mandatory	Additional info
gcp	credentials_path		no	Service account JSON used by Lithops on your machine and to select the VM service account
gcp	region		no	GCP region. Derived from zone if omitted

GCE - Create and Reuse Modes

Group	Key	Default	Mandatory	Additional info
gcp_compute_engine	project_name		yes	GCP project ID
gcp_compute_engine	zone		yes	Compute Engine zone, for example us-east1-b
gcp_compute_engine	region	derived from zone	no	Region used for subnet and NAT
gcp_compute_engine	service_account		no	Service account email attached to VMs. Default: client_email from credentials_path
gcp_compute_engine	network_name		no	Existing VPC name. If not provided, Lithops creates a new network
gcp_compute_engine	subnet_name		no	Existing subnet name when using a custom VPC
gcp_compute_engine	source_image	ubuntu-2404-lts-amd64	no	Boot image reference
gcp_compute_engine	master_instance_type	e2-small	no	Master VM machine type
gcp_compute_engine	worker_instance_type	e2-standard-2	no	Worker VM machine type
gcp_compute_engine	ssh_username	ubuntu	no	Username to access the VM
gcp_compute_engine	ssh_password		no	Password for worker VMs. If not provided, it is created randomly
gcp_compute_engine	ssh_key_filename		no	SSH private key for the master VM. If not provided, Lithops creates ~/.ssh/lithops-key-<id>.gcp_ce.id_rsa
gcp_compute_engine	request_spot_instances	False	no	Use Spot VMs for workers
gcp_compute_engine	delete_on_dismantle	True	no	Delete worker VMs when stopped. Master VM is never deleted when stopped
gcp_compute_engine	max_workers	100	no	Max number of workers per FunctionExecutor()
gcp_compute_engine	worker_processes	AUTO	no	Parallel Lithops processes per worker. Default: CPUs of worker_instance_type
gcp_compute_engine	runtime	python3	no	Runtime name. Default: python3 on the VM
gcp_compute_engine	auto_dismantle	True	no	If False, VMs are not stopped automatically
gcp_compute_engine	soft_dismantle_timeout	300	no	Seconds to stop the VM after a job completed
gcp_compute_engine	hard_dismantle_timeout	3600	no	Seconds to stop the VM after a job started
gcp_compute_engine	exec_mode	reuse	no	One of: consume, create or reuse
gcp_compute_engine	extra_apt_packages	[]	no	Extra apt packages on master/worker VMs during setup
gcp_compute_engine	extra_python_packages	[]	no	Extra pip packages on master/worker VMs after Lithops

Consume mode

In this mode, Lithops uses an existing VM. The VM must be reachable by SSH and have a service account with GCS access.

Configuration

lithops:
  backend: gcp_compute_engine

gcp:
  credentials_path: <FULL_PATH_TO_SERVICE_ACCOUNT_JSON>

gcp_compute_engine:
  exec_mode: consume
  project_name: <GCP_PROJECT_ID>
  zone: <ZONE>
  instance_name: <EXISTING_VM_NAME>

Summary of configuration keys for the consume mode

Group	Key	Default	Mandatory	Additional info
gcp_compute_engine	instance_name		yes	Existing VM instance name
gcp_compute_engine	project_name		yes	GCP project ID
gcp_compute_engine	zone		yes	Compute Engine zone
gcp_compute_engine	ssh_username	ubuntu	no	Username to access the VM
gcp_compute_engine	ssh_key_filename		no	Path to the SSH private key. If not provided, Lithops creates ~/.ssh/lithops-key-<id>.gcp_ce.id_rsa
gcp_compute_engine	worker_processes	AUTO	no	Parallel Lithops processes per worker

Test Lithops

Once you have your compute and storage backends configured, you can run a Hello World function with:

lithops hello -b gcp_compute_engine -s gcp_storage

Viewing the execution logs

You can view the function execution logs on your local machine using the Lithops client:

lithops logs poll

VM Management

Lithops for GCE follows a master-worker architecture (1:N).

All VMs, including the master, are automatically stopped after a configurable timeout (see hard/soft dismantle timeouts). Stopped master and worker VMs are started again on the next job in reuse mode.

You can open an SSH session to the master VM with:

lithops attach -b gcp_compute_engine

The master and worker VMs store Lithops service logs in /tmp/lithops-root/*-service.log.

To list available workers:

lithops worker list -b gcp_compute_engine

To list submitted jobs:

lithops job list -b gcp_compute_engine

To delete workers only:

lithops clean -b gcp_compute_engine -s gcp_storage

To delete workers, the master VM, and Lithops-created network resources:

lithops clean -b gcp_compute_engine -s gcp_storage --all

Architecture diagram

        flowchart TB
  subgraph gcp [GCP project / region]
    NET["VPC lithops-net-XXXXXX"]
    SUB["Subnet lithops-net-XXXXXX-subnet"]
    FW1["Firewall SSH :22 from internet"]
    FW2["Firewall internal :8080/8081/6379/22"]
    NAT["Cloud Router + NAT lithops-net-XXXXXX-router"]
    M["Master VM lithops-master-XXXXXX\n+ ephemeral external IP"]
    W["Worker VMs lithops-worker-*\nprivate IP only"]
  end
  LAPTOP["Your laptop"] -->|SSH :22| M
  M -->|private network| W
  W -->|egress| NAT
  NAT --> INTERNET[Internet apt/docker/pip]
  M --> SUB
  W --> SUB
  SUB --> NET