Command Line Interface Reference

This page provides a comprehensive reference for the Cloud Tasks command-line interface, cloud_tasks. The general format of a command is:

cloud_tasks <command> <options>

To get a list of available commands, run:

cloud_tasks --help

To get a list of options for a specific command, run:

cloud_tasks <command> --help

Since many commands take similar options, we will start by listing the shared options and then reference them as needed.

Command Line Options

Common Options

All commands support these common options:

--config CONFIG

Path to configuration file (optional if no configuration file is needed)

--provider PROVIDER

Cloud provider (aws or gcp), overrides configuration file

--verbose, -v

Enable verbose output (-v for INFO, -vv for DEBUG, default is WARNING)

Job-Specific Options

In addition to the common options, each job-specific command has additional options that specify job-related information. They override any options in the configuration file (see Provider-Specific Options).

--job-id JOB_ID

A unique identifier for the job

--queue-name QUEUE_NAME

The name of the task queue to use (derived from job ID if not provided; only use this in special circumstances)

--region REGION

The region to use (derived from zone if not provided)

--zone ZONE

The zone to use (if not specified, all zones in a region will be used)

--exactly-once-queue

If specified, task and event queue messages are guaranteed to be delivered exactly once to any recipient

--no-exactly-once-queue

If specified, task and event queue messages are delivered at least once, but could be delivered multiple times

Provider-Specific Options

In addition to the common options, each provider has additional options that are specific to that provider. They override any options in the configuration file (see Provider-Specific Options).

AWS

--access-key ACCESS_KEY

The access key to use

--secret-key SECRET_KEY

The secret key to use

GCP

--project-id PROJECT_ID

The ID of the project to use [Required for most operations]

--credentials-file CREDENTIALS_FILE

The path to a file containing the credentials to use; if not specified, the default credentials will be used

--service-account SERVICE_ACCOUNT

The service account to use; required for worker processes on cloud-based instances to have access to system resources [Required when creating instances]

Instance Type Selection Options

These options are used to constrain the instance types. They override any constraints in the configuration file (see Options to select a compute instance type).

--architecture ARCHITECTURE

The architecture to use; valid values are X86_64 and ARM64 (defaults to X86_64)

--cpu-family CPU_FAMILY

The CPU family to use, for example Intel Cascade Lake or AMD Genoa.

--min-cpu-rank MIN_CPU_RANK

The minimum CPU performance rank to use (0 is the slowest)

--max-cpu-rank MAX_CPU_RANK

The maximum CPU performance rank to use (0 is the slowest)

--instance-types TYPES

A single instance type or list of instance types to use; instance types are specified using Python-style regular expressions (if no anchor character like ^ or $ is specified, the given string will match any part of the instance type name)

--min-cpu N

The minimum number of vCPUs per instance

--max-cpu N

The maximum number of vCPUs per instance

--cpus-per-task N

The number of vCPUs per task; this is also used to configure the worker process to limit the number of tasks that can be run simultaneously on a single instance

--min-tasks-per-instance N

The minimum number of tasks per instance

--max-tasks-per-instance N

The maximum number of tasks per instance

--min-total-memory N

The minimum amount of memory in GB per instance

--max-total-memory N

The maximum amount of memory in GB per instance

--min-memory-per-cpu N

The minimum amount of memory per vCPU

--max-memory-per-cpu N

The maximum amount of memory per vCPU

--min-memory-per-task N

The minimum amount of memory per task

--max-memory-per-task N

The maximum amount of memory per task

--min-local-ssd N

The minimum amount of local extra SSD storage in GB per instance

--max-local-ssd N

The maximum amount of local extra SSD storage in GB per instance

--local-ssd-base-size N

The base size of the local extra SSD storage in GB per instance

--min-local-ssd-per-cpu N

The minimum amount of local extra SSD storage per vCPU

--max-local-ssd-per-cpu N

The maximum amount of local extra SSD storage per vCPU

--min-local-ssd-per-task N

The minimum amount of local extra SSD storage per task

--max-local-ssd-per-task N

The maximum amount of local extra SSD storage per task

--total-boot-disk-size N

The total size of the boot disk in GB per instance

--boot-disk-base-size N

The base size of the boot disk in GB per instance

--boot-disk-per-cpu N

The amount of boot disk per vCPU

--boot-disk-per-task N

The amount of boot disk per task

--boot-disk-types TYPES

The types of boot disks to use

--boot-disk-iops N

The number of provisioned IOPS for the boot disk, if applicable

--boot-disk-throughput N

The number of provisioned throughput in MB/s for the boot disk, if applicable

Number of Instances Options

These options are used to constrain the number of instances. They override any constraints in the configuration file (see Options to constrain the number of instances).

--min-instances N

The minimum number of instances to use (defaults to 1)

--max-instances N

The maximum number of instances to use (defaults to 10)

--min-total-cpus N

The minimum total number of vCPUs to use

--max-total-cpus N

The maximum total number of vCPUs to use

--cpus-per-task N

The number of vCPUs per task; this is also used to configure the worker process to limit the number of tasks that can be run simultaneously on a single instance

--min-tasks-per-instance N

The minimum number of tasks per instance

--max-tasks-per-instance N

The maximum number of tasks per instance

--min-simultaneous-tasks N

The minimum number of tasks to run simultaneously

--max-simultaneous-tasks N

The maximum number of tasks to run simultaneously

--min-total-price-per-hour N

The minimum total price per hour to use

--max-total-price-per-hour N

The maximum total price per hour to use (defaults to 10)

VM Options

These options are used to specify the type of VM to use. They override any options in the configuration file (see Options to specify the type of VM).

--use-spot

Use spot instances instead of on-demand instances

Boot Options

These options are used to specify the boot process. They override any options in the configuration file (see Options to specify the boot process).

--startup-script-file FILE

The path to a file containing the startup script

--image IMAGE

The image to use for the VM

Worker and Run Options

These options are used to specify the worker and run process behavior. They override any options in the configuration file (see Options to specify the worker process and run process).

--scaling-check-interval SECONDS

The interval to check for scaling opportunities (defaults to 60)

--instance-termination-delay SECONDS

The delay to wait before terminating an instance (defaults to 60)

--max-runtime SECONDS

The maximum runtime for a task (defaults to 60)

--retry-on-exit

If specified, tasks will be retried if the worker exits prematurely, e.g. due to a crash

--no-retry-on-exit

If specified, tasks will not be retried if the worker exits prematurely, e.g. due to a crash (default)

--retry-on-exception

If specified, tasks will be retried if the user function raises an unhandled exception

--no-retry-on-exception

If specified, tasks will not be retried if the user function raises an unhandled exception (default)

--retry-on-timeout

If specified, tasks will be retried if they exceed the maximum runtime specified by –max-runtime

--no-retry-on-timeout

If specified, tasks will not be retried if they exceed the maximum runtime specified by –max-runtime (default)

Information Commands

list_regions

List available regions, and optionally availability zones and other details, for a provider.

cloud_tasks list_regions
  [Common options]
  [Provider-specific options]
  [Additional options]

Additional options:

--prefix PREFIX

Filter regions by name prefix

--zones

Show availability zones for each region

--detail

Show additional provider-specific information

Examples:

AWSGCP

XXX Update
$ cloud_tasks list_regions --provider aws --detail --zones --prefix us-west
Found 2 regions (filtered by prefix: us-west)

Region                    Description
----------------------------------------------------------------------------------------------------
us-west-1                 AWS Region us-west-1
Availability Zones: us-west-1a, us-west-1b
Opt-in Status: opt-in-not-required

us-west-2                 AWS Region us-west-2
Availability Zones: us-west-2a, us-west-2b, us-west-2c, us-west-2d
Opt-in Status: opt-in-not-required

list_images

List available VM images.

cloud_tasks list_images
  [Common options]
  [Provider-specific options]
  [Additional options]

Additional options:

--user

Include user-created images; otherwise, only include system-provided public images

--filter TEXT

Include only images containing TEXT in any field

--sort-by FIELDS

Sort the result by one or more comma-separated fields; available fields are family, name, project, source. Prefix with - for descending order. Partial field names like fam for family or proj for project are supported.

--limit N

Limit the number of results to the first N after sorting

--detail

Show detailed information

Examples:

AWSGCP

XXX Update
$ cloud_tasks list_images --provider aws --filter sapcal --detail --sort-by=-name --limit 2
Retrieving images...
Found 2 filtered images for aws:

Name                                                                             Source
------------------------------------------------------------------------------------------
suse-sles-15-sp6-sapcal-v20250409-hvm-ssd-x86_64                                 AWS
SUSE Linux Enterprise Server 15 SP6 for SAP CAL (HVM, 64-bit, SSD Backed)
ID: ami-09b43f66ab9cce59a
CREATION DATE: 2025-04-09T21:15:49.000Z    STATUS: available
URL: N/A

suse-sles-15-sp6-sapcal-v20250130-hvm-ssd-x86_64                                 AWS
SUSE Linux Enterprise Server 15 SP6 for SAP CAL (HVM, 64-bit, SSD Backed)
ID: ami-013778510a6146053
CREATION DATE: 2025-01-31T12:06:46.000Z    STATUS: available
URL: N/A


To use a custom image with the 'run' command, use the --image parameter.
For AWS, specify the AMI ID: --image ami-12345678

list_instance_types

List available instance types with pricing.

cloud_tasks list_instance_types
  [Common options]
  [Provider-specific options]
  [Instance type selection options]
  [VM options]
  [Additional options]

Additional options:

--filter TEXT

Include only images containing TEXT in any field

--sort-by FIELDS

Sort the result by one or more comma-separated fields; available fields are name, vcpu, mem, local_ssd, storage, vcpu_price, mem_price, local_ssd_price, boot_disk_price, boot_disk_iops_price, boot_disk_throughput_price, price_per_cpu, mem_per_gb_price, local_ssd_per_gb_price, boot_disk_per_gb_price, total_price, total_price_per_cpu, zone, processor_type, performance_rank, description. Prefix with - for descending order. Partial field names like ram or mem for mem_gb or v for vcpu are supported.

--limit N

Limit the number of results to the first N after sorting

--detail

Show detailed information

Examples:

AWSGCP

XXX Update
$ cloud_tasks list_instance_types --provider aws --region us-west-1 --instance-types "m4.*" --sort-by=-cpu,-mem --limit 5
Retrieving instance types...
Retrieving pricing information...

Instance Type                  Arch vCPU   Mem (GB)  LSSD (GB)  Disk (GB)  Total $/Hr         Zone
-----------------------------------------------------------------------------------------------------------
m4.16xlarge                  x86_64   64      256.0          0          0     $3.7440  us-west-1-*
m4.10xlarge                  x86_64   40      160.0          0          0     $2.3400  us-west-1-*
m4.4xlarge                   x86_64   16       64.0          0          0     $0.9360  us-west-1-*
m4.2xlarge                   x86_64    8       32.0          0          0     $0.4680  us-west-1-*
m4.xlarge                    x86_64    4       16.0          0          0     $0.2340  us-west-1-*

Job Management Commands

run

The run command handles the complete task execution workflow in a single command. It automates: queue management, task loading, instance orchestration, event monitoring, and cleanup.

During job execution, the status of each task is tracked in a SQLite database and reported to the console. The name of the database file is derived from the job ID or can be specified with the --db-file option. If the job run is interrupted, the database file is saved and job execution can be resumed by using the --continue option based on the contents of the database file.

Fresh Run Mode (default): Deletes existing SQLite database and queues, creates new ones, loads tasks into the database and cloud queue, manages instances, monitors events, and automatically terminates instances and deletes queues when all tasks complete. If the existing queue has messages, the user will be prompted for confirmation unless --force is specified.

Continue Mode (--continue): Resumes from a previous interrupted run by reading the SQLite database, draining pending events, and continuing to manage instances until completion.

cloud_tasks run
  [Common options]
  [Provider-specific options]
  [Job-specific options]
  [Instance type selection options]
  [Number of instances options]
  [VM options]
  [Boot options]
  [Worker and Manage Pool options]
  [Additional options]

Additional options:

--task-file TASK_FILE

Path to task file (JSON or YAML); required for fresh runs, not used with –continue

--start-task N

Skip tasks until this task number (1-based)

--limit N

Maximum number of tasks to enqueue

--max-concurrent-queue-operations N

Maximum concurrent tasks to enqueue (default: 100)

--continue

Resume from a previous interrupted run using the existing SQLite database and cloud state

--db-file DB_FILE

Path to SQLite database file (default: {job_id}.db); can also be set in configuration file under run.db_file

--output-file OUTPUT_FILE

Optional file to write events to in JSON-lines format in addition to the SQLite database

--force, -f

Force fresh run without confirmation even if queue has existing messages

--dry-run

Do not actually load any tasks or create or delete any instances

SQLite Database: The database file (default: {job_id}.db) persistently tracks task status and events. It contains:

• tasks table: task_id, task_data, status, retry flag, timestamps, hostname, results, exceptions, exit codes

• events table: Raw event log from workers

Task Completion: A task is considered complete when it has any status with retry=False.

Keyboard Interrupt (Ctrl+C): When interrupted, you are prompted:

Received interrupt.

Choose action:
  [T] Terminate all instances and delete queues
  [L] Leave instances running (can resume with --continue)
  [C] Cancel and continue running

Enter choice (T/L/C):

Final Report: Upon completion, a comprehensive report is printed with:

• Task counts by status

• Total elapsed time and throughput (tasks/hour)

• Task execution time statistics (range, mean, median, 90th/95th percentile)

• Exception summaries with counts

• Spot termination tracking

Examples:

AWSGCP

# Fresh run
cloud_tasks run --config myconfig.yml --task-file tasks.json

# Resume after crash
cloud_tasks run --config myconfig.yml --continue

# With event logging
cloud_tasks run --config myconfig.yml --task-file tasks.json --output-file events.json

monitor_event_queue

Monitor the event queue and update the task database. This command is useful when you are running workers locally (not using cloud-managed compute instances) but still want automated event monitoring and SQLite-based task tracking. Note this does not manage compute instances.

cloud_tasks monitor_event_queue
  [Common options]
  [Job-specific options]
  [Additional options]

Additional options:

--db-file DB_FILE

Path to SQLite database file (default: {job_id}.db)

--output-file OUTPUT_FILE

Optional file to write events to in JSON-lines format

--print-events

Print events to stdout as they are received

--no-auto-complete

Don’t stop automatically when all tasks complete

Use Case:

This command is designed for scenarios where you want to run workers manually (e.g., for local testing or debugging) but still use the automated event monitoring and task tracking infrastructure.

Workflow:

1. Create queues and database (without starting cloud instances):

   cloud_tasks run --config config.yml --task-file tasks.json --dry-run
   

2. Start workers locally in separate terminals:

   # Terminal 1
   python your_worker.py --config config.yml
   
   # Terminal 2
   python your_worker.py --config config.yml
   

3. Monitor progress in another terminal:

   cloud_tasks monitor_event_queue --config config.yml
   

Behavior:

• Opens an existing SQLite database (created by run --dry-run or a previous run)

• Monitors the event queue for task completion events from workers

• Updates the database with task status, results, and statistics

• Prints periodic status summaries to the console

• Automatically stops when all tasks complete (unless --no-auto-complete is specified)

• Optionally writes events to a JSON-lines output file for archival

Comparison with ``run`` command:

• run: Creates and manages cloud instances + monitors events (full automation)

• monitor_event_queue: Only monitors events (for manual worker management)

Examples:

AWSGCP

# Monitor with default settings
cloud_tasks monitor_event_queue --config config.yml

# Monitor with output file
cloud_tasks monitor_event_queue --config config.yml --output-file events.jsonl

# Monitor indefinitely (don't stop when tasks complete)
cloud_tasks monitor_event_queue --config config.yml --no-auto-complete

status

Check the status of a running job.

cloud_tasks status
  [Common options]
  [Provider-specific options]
  [Job-specific options]

Examples:

AWSGCP

TODO

stop

Stop a job and terminate its instances.

cloud_tasks stop
  [Common options]
  [Provider-specific options]
  [Job-specific options]
  [Additional options]

Additional options:

--purge-queue

Purge the task queue after stopping instances

Examples:

AWSGCP

TODO

list_running_instances

List currently running instances. By default only active instances created by Cloud Tasks are shown. If only a region is specified, instances in all zones in that region are shown. If a zone is specified, only instances in that zone are shown.

cloud_tasks list_running_instances
  [Common options]
  [Provider-specific options]
  [Additional options]

Additional options:

--job-id JOB_ID

Filter by job ID

--all-instances

Show all instances including ones that were not created by Cloud Tasks

--include-terminated

Include terminated instances

--sort-by FIELDS

Sort results by comma-separated fields (e.g., “state,type” or “-created,id”). Available fields: id, type, state, zone, creation_time. Prefix with “-” for descending order. Partial field names like “t” for “type” or “s” for “state” are supported.

--detail

Show detailed information

Examples:

AWSGCP

TODO

Queue Management Commands

load_queue

Load tasks into the local SQLite database and cloud task queue without starting any compute instances. This allows you to separate the task loading step from the run command (e.g. load once, then run with --continue to manage instances only).

Fresh load (default): Deletes the existing SQLite database and cloud queues (after optional confirmation if the queue has messages), creates new queues, loads tasks from the task file into the database and enqueues them to the cloud task queue.

Continue mode (--continue): Opens the existing SQLite database and prints task statistics and queue depth only. No task file is read and no messages are enqueued.

cloud_tasks load_queue
  [Common options]
  [Job-specific options]
  [Provider-specific options]
  [Additional options]

Additional options:

--task-file TASK_FILE

Path to task file (JSON or YAML); required unless –continue

--start-task N

Skip tasks until this task number (1-based)

--limit N

Maximum number of tasks to enqueue

--max-concurrent-queue-operations N

Maximum concurrent queue operations (default: 100)

--db-file DB_FILE

Path to SQLite database file (default: {job_id}.db)

--continue

Open existing database and show status only (no load)

--force, -f

Force fresh load without confirmation even if queue has existing messages

Examples:

AWSGCP

$ cloud_tasks load_queue --provider aws --job-id my-job --task-file examples/parallel_addition/addition_tasks.json
Loaded 10000 tasks. Queue depth (may be approximate): 10000

$ cloud_tasks load_queue --config myconfig.yml --continue
Database: my-job.db (10000 tasks)
Status: ...

show_queue

Show information about a task queue. Note that some providers do not provide an accurate count of messages remaining in a queue.

cloud_tasks show_queue
  [Common options]
  [Job-specific options]
  [Provider-specific options]
  [Additional options]

Additional options:

--detail

Show a sample message

Examples:

AWSGCP

$ cloud_tasks show_queue --provider aws --job-id my-job --detail
Checking queue depth for 'my-job'...
Current depth: 10000 message(s)

Attempting to peek at first message...

--------------------------------------------------
SAMPLE MESSAGE
--------------------------------------------------
Task ID: addition-task-000035
Receipt Handle: AQEBt0nqkqnpbta3H0OV62eJGwx6do5rXY8MW+NbGlnhwE0Etz...

Data:
{
"num1": 1438,
"num2": 49332
}

Note: Message was not removed from the queue.

purge_queue

Remove all messages from the task and event queues. This allows you to start fresh by loading new tasks.

cloud_tasks purge_queue
  [Common options]
  [Job-specific options]
  [Provider-specific options]
  [Additional options]

Additional options:

--task-queue-only

Purge only the task queue (not the event queue)

--event-queue-only

Purge only the event queue (not the task queue)

--force

Purge without confirmation

Examples:

AWSGCP

XXX Update this
$ cloud_tasks purge_queue --provider aws --job-id my-job

WARNING: This will permanently delete all 10000+ messages from queue 'my-job' on 'AWS'.
Type 'EMPTY my-job' to confirm: EMPTY my-job
Emptying queue 'my-job'...
Queue 'my-job' has been emptied. Removed 10000+ message(s).

delete_queue

Delete the task and event queues and their infrastructure. This permanently frees up the resources used by the queues. Only do this if there are no processes running that use the queues.

cloud_tasks delete_queue
  [Common options]
  [Job-specific options]
  [Provider-specific options]
  [Additional options]

Additional options:

--task-queue-only

Delete only the task queue (not the event queue)

--event-queue-only

Delete only the event queue (not the task queue)

--force

Delete without confirmation

Examples:

AWSGCP

$ cloud_tasks delete_queue --provider aws --job-id my-job

WARNING: This will permanently delete the queue 'my-job' from AWS.
This operation cannot be undone and will remove all infrastructure.
Type 'DELETE my-job' to confirm: DELETE my-job
Deleting queue 'my-job' from AWS...
Queue 'my-job' has been deleted.

WARNING: This will permanently delete the queue 'my-job-results' from AWS.
This operation cannot be undone and will remove all infrastructure.
Type 'DELETE my-job-results' to confirm: DELETE my-job-results
Deleting queue 'my-job-results' from AWS...
Queue 'my-job-results' has been deleted.

Exit Status

The CLI returns the following exit codes:

• 0 - Success

• 1 - Error occurred during command execution