7. Clustering

In Koios Automation Platform, automation controller runs as two Kubernetes deployments that the Koios operator manages:

  • web pods serve the controller API behind the platform gateway;

  • task pods run the dispatcher, the callback receiver and the other background services. Each task pod is a control instance of the controller cluster: it registers itself when it starts.

Playbook jobs run separately from the controller services. By default they run as pods of their own, in the namespace of the platform, through the default container group (see Instance Groups). The controlplane instance group holds the control instances, which run project updates and other internal work. A new platform has one control instance, and the cluster grows by adding task pods.

Controller web and task pods share the controller’s PostgreSQL database. The operator can deploy the database or connect the controller to an external database (see KoiosPlatform). Other platform services use their own databases, even when the platform-managed PostgreSQL server hosts them together.

The deployment relationships are:

Component

Responsibility

Deployment relationship

Platform gateway

Jewel handles platform identity and access; Envoy routes public API requests to the appropriate service.

Separate from controller web and task Deployments; it is the public entry point.

Controller web pods

Serve the controller API behind the gateway.

Replicas of the controller web Deployment.

Controller task pods

Run scheduling, dispatch, callback processing and control-plane Receptor services. Internal work such as project updates can run in their control-plane execution environment container.

Replicas of the controller task Deployment; each registers a control instance.

Controller local Redis/Valkey

Provides local controller queueing and communication within each web or task pod.

A sidecar in each controller pod, not a Redis process installed on each Kubernetes host.

Platform shared Redis/Valkey

Supplies the Redis-compatible service used by gateway and other configured platform components.

Separate from the controller’s local sidecars; managed or externally supplied according to the platform configuration.

PostgreSQL

Persists service data, including controller job and event records.

Platform-managed database server or external database configuration.

Job execution

Runs the selected playbook in its execution environment.

Container group job pods by default; external execution nodes are also supported through Receptor.

Increasing controller replicas does not duplicate a complete platform stack on each Kubernetes node. Pod placement, execution capacity and availability of the database, Redis-compatible services and gateway must be considered separately.

7.1. Scaling the web and task pods

Set the replicas of the controller under spec.controller of the KoiosPlatform; the operator passes these fields to the controller deployments:

  • replicas scales both the web and the task pods to the same number;

  • web_replicas and task_replicas set each deployment on its own, and take precedence over replicas.

The operator manages the replicas while web_manage_replicas and task_manage_replicas are true, the default; set them to false to scale the deployments yourself, for example with a HorizontalPodAutoscaler.

For example, two web pods and three task pods (three control instances):

kubectl patch koiosplatform koios -n koios --type merge \
  -p '{"spec": {"controller": {"web_replicas": 2, "task_replicas": 3}}}'

New task pods register as control instances and join the controlplane group on their own.

7.2. Assigning automation controller pods to specific nodes

The same fields constrain where the controller pods run. Each one applies to both deployments, and its web_ and task_ variants to one of them:

Field (under spec.controller)

Applies to

node_selector

nodeSelector of the web and task pods

web_node_selector, task_node_selector

nodeSelector of one deployment

tolerations

Tolerations of the web and task pods

web_tolerations, task_tolerations

Tolerations of one deployment

affinity

Affinity rules of the web and task pods

web_affinity, task_affinity

Affinity rules of one deployment

topology_spread_constraints

topologySpreadConstraints of the web and task pods

web_topology_spread_constraints, task_topology_spread_constraints

topologySpreadConstraints of one deployment

annotations

Annotations of the web and task pods

node_selector, tolerations, topology_spread_constraints, annotations and their variants take YAML as a string; affinity takes an object. A web_ or task_ variant replaces the common value for its deployment; it is not merged with it. For example, task pods on dedicated nodes:

spec:
  controller:
    task_node_selector: |
      node-role.example.com/automation: "true"
    task_tolerations: |
      - key: dedicated
        operator: Equal
        value: automation
        effect: NoSchedule

The pods of jobs are placed by their container group: set a node selector or tolerations in its pod specification (see Customize the pod spec). The gateway, Event-Driven Ansible, automation hub and PostgreSQL have their own placement fields in the KoiosPlatform (see KoiosPlatform).

7.3. Status and monitoring

The controller reports the state of its cluster at /api/controller/v2/ping/, without authentication: the node that served the request, registered non-hop instances with their last heartbeat, and the instance groups. More detail, such as running jobs and capacity, is at /api/controller/v2/instances/ and /api/controller/v2/instance_groups/, and in the user interface under Automation Execution > Infrastructure, in Instances and Instance Groups.

The Instances page with the control instance of a new platform, its task pod.

For the state of the deployment itself, look at the KoiosPlatform: its conditions say whether its last reconciliation succeeded and, if not, why (see Operate).

7.4. Failure behavior

Each task pod runs several services: the dispatcher, which runs the work; the callback receiver, which stores the events of running jobs; and the services that keep the instance registered. If they keep failing, the health checks of the task container fail and Kubernetes restarts it.

When a control instance stops sending heartbeats for long enough that another instance declares it lost, the other instances take over:

  • the jobs it was running are marked as failed, with the explanation Job reaped due to instance shutdown;

  • the jobs it had not started yet go back to the queue, for another instance;

  • the instance is removed from the cluster, and the restarted pod registers again.

An instance that restarts before it is declared lost keeps its identity and picks up its own work.

Run at least two task pods so that losing one does not stop new work.

7.5. Job runtime behavior

When a job is launched, it is queued, and a control instance with capacity picks it up and starts it in the instance group the job runs in (by default, the default container group). The job’s pod runs the playbook in its execution environment and streams its events back.

Projects are synchronized before they are used: if the revision a job needs is not there yet, it is fetched first, and the sync is recorded as a project update with launch_type = sync. Collections and roles from Galaxy or automation hub are downloaded in that update, which takes space in temporary storage; for large projects, allow for it in the resources of the task pods.

The request and result paths are separate:

  1. A user or API client reaches the platform gateway, which routes controller requests under /api/controller/ to the controller web service.

  2. The controller schedules the job and a task pod dispatches it through Receptor to the selected execution location.

  3. The execution worker runs the playbook in its execution environment and returns job results to the controller.

  4. Controller callback processing stores job events in PostgreSQL; users read status and output through the gateway and controller API.

/api/controller/v2/ping/ reports controller cluster information. It is an observation endpoint, not a load balancer distributing requests across complete HTTP/worker/Redis host stacks.

7.5.1. Job runs

By default, a job can be picked up by any control instance, and it runs in the instance group set on its job template, inventory or organization (see Control Where a Job Runs).

To take an instance out of rotation, disable it (enabled: false) in the API or the user interface. No new jobs are assigned to it, and the jobs it is running finish normally.