9. Managing Capacity With Instances

Open Automation Execution > Infrastructure > Instances to inspect controller instances and register external execution or hop nodes. Registering a node creates its controller record; it does not install or configure the remote machine. The node is outside the Kubernetes cluster and is provisioned separately with its install bundle. Scale controller pods through the KoiosPlatform, not by creating control instances in this form.

Note

In Koios Automation Platform, jobs run by default in the default container group, inside the cluster, and no mesh is needed. Validated in Koios: execution nodes outside the cluster on Kubernetes (Rocky Linux 9 and Debian 13, see Preparing the node) and on OpenShift (Rocky Linux 9), the default execution environment from a private registry, and the mesh ingress on OpenShift. A separate hop node outside the cluster has not been tested yet. The Koios operator installs the AWXMeshIngress resource type and reconciles the mesh ingresses you create.

Instances serve as nodes in your mesh topology. Automation mesh allows you to extend the footprint of your automation. Where you launch a job and where the ansible-playbook runs can be in different locations.

Site A pointing to Site B and dotted arrows to two hosts from Site B

Automation mesh is useful for:

  • traversing difficult network topologies

  • bringing execution capabilities (the machine running ansible-playbook) closer to your target hosts

The nodes (control, hop, and execution instances) are interconnected via receptor, forming a virtual mesh.

Control node pointing to hop node, which is pointing to two execution nodes.

9.1. Prerequisites

  • A Red Hat family (RHEL 9 and compatible) or Debian operating system, prepared as described in Preparing the node. Bring a machine online with that OS. Use a stable IP address or DNS hostname. Name resolution and connectivity must work in the direction of each peer connection. If control nodes connect directly to this machine, they must be able to reach its listener port (for example, 27199). A node using mesh ingress can instead initiate the connection towards the cluster.

    In general, the more CPU cores and memory the machine has, the more jobs that can be scheduled to run on that machine at once. See Automation controller Capacity Determination and Job Impact for more information on capacity.

  • The system that is going to run the install bundle to setup the remote node requires the collection ansible.receptor to be installed:

    • If machine has access to the internet:

    ansible-galaxy collection install -r requirements.yml
    

    Installing the receptor collection dependency from the requirements.yml file will consistently retrieve the receptor version specified there, as well as any other collection dependencies that may be needed in the future.

  • Use a platform administrator account with controller system administrator permissions to create, edit or remove instances. System auditors can inspect instances and existing health-check results; starting a health check requires controller system administrator permission. Auditor access does not grant permission to change instances.

  • The execution environment that jobs use on the node must be pullable from it. Images in a private registry, such as the default execution environment of Koios Automation Platform, need a Container Registry credential on the execution environment: an external node pulls with Podman and uses the credential of the execution environment rather than the cluster namespace’s pull secrets (see Using a private image for the default EE).

9.1.1. Preparing the node

Execution nodes have been validated in Koios with Rocky Linux 9 and Debian 13. The preparation below applies only to the external execution machine, not to a controller host or cluster node. Run provisioning from a separate administration environment with the required Ansible tools and privileges. Each adjustment below is shown with the error it avoids.

Primary group of the receptor user (bundles from before 2026.11.6). Since 2026.11.6 the bundle gives the awx user its own awx primary group, and nothing needs to be done. Bundles downloaded from older releases created the user without naming its group, and it could end up with another primary group (users on both Rocky Linux 9 and Debian 13); rootless podman then fails in every job with newuidmap: Target process is owned by a different user. On a node installed with such a bundle, download the bundle again and run it, or check it with id awx and, if the primary group is not awx, fix it with receptor stopped:

sudo systemctl stop receptor
sudo usermod -g awx awx
sudo -u awx -H podman system migrate
sudo systemctl start receptor

On Debian (validated with Debian 13; other releases are expected to need the same, but have not been tested), the install bundle and the way jobs run on a node, written for the Red Hat family, also need the following.

Python packages from pip. The bundle installs ansible-runner and receptorctl with the system pip3, which Debian blocks (error: externally-managed-environment). Allow it only while the bundle runs: if /etc/pip.conf does not exist yet, create it before running the bundle, and remove it afterwards (repeat this whenever you run the bundle again):

printf '[global]\nbreak-system-packages = true\n' | sudo tee /etc/pip.conf
# ... run the install bundle ...
sudo rm /etc/pip.conf

CA trust paths. The Paths to expose to isolated jobs setting (AWX_ISOLATION_SHOW_PATHS, under Settings > Automation Execution > Job) can expose the execution host’s /etc/pki/ca-trust and /usr/share/pki to the execution environment of jobs that run on external nodes, when those paths are configured. Those paths do not exist on Debian, and jobs fail with mounting overlay failed “/etc/pki/ca-trust”. There are two ways out:

  • If the execution environment carries the certificate authorities your jobs need, remove those two paths from the setting. This is a change for every node of the controller, so check it against the other nodes.

  • To keep using the node’s certificate authorities, recreate the Red Hat layout with Debian’s bundle (do not create the directories empty: they would hide the execution environment’s own certificate authorities):

    sudo install -d -m 0755 /etc/pki/ca-trust/extracted/pem /etc/pki/ca-trust/extracted/openssl /usr/share/pki
    sudo install -m 0644 /etc/ssl/certs/ca-certificates.crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
    sudo install -m 0644 /etc/ssl/certs/ca-certificates.crt /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
    

    Repeat the copies after update-ca-certificates when you add certificate authorities to the node.

9.2. Common topologies

Instances make up the network of devices that communicate with one another. They are the building blocks of an automation mesh. These building blocks serve as nodes in a mesh topology. There are several kinds of instances:

Node Type

Description

Control

Nodes that run persistent automation controller services, and delegate jobs to hybrid and execution nodes

Hybrid

Nodes that run persistent automation controller services and execute jobs (not applicable to operator-based installations)

Hop

Used for relaying across the mesh only

Execution

Nodes that run jobs delivered from control nodes (jobs submitted from the user’s Ansible automation)

9.2.1. Simple topology

One of the ways to expand job capacity is to create a standalone execution node that can be added to run alongside the Kubernetes deployment of automation controller. These machines will not be a part of the automation controller Kubernetes cluster. The control nodes running in the cluster will connect and submit work to these machines via Receptor. The machines are registered in automation controller as type “execution” instances, meaning they will only be used to run automation controller jobs, not dispatch work or handle web requests as control nodes do.

Hop nodes can be added to sit between the control plane of automation controller and standalone execution nodes. These machines will not be a part of the automation controller Kubernetes cluster and they will be registered in automation controller as node type “hop”, meaning they will only handle inbound and outbound traffic for otherwise unreachable nodes in a different or more strict network.

Below is an example of an automation controller task pod with two execution nodes. Traffic to execution node 2 flows through a hop node that is setup between it and the control plane.

Automation controller task pod with a hop node between the control plane of automation controller and standalone execution nodes.

Below are sample values used to configure each node in a simple topology:

Instance type

Hostname

Listener port

Peers from control nodes

Peers

Control plane

awx-task-65d6d96987-mgn9j

n/a

n/a

[hop node]

Hop node

awx-hop-node

27199

True

[]

Execution node

awx-example.com

n/a

False

[hop node]

9.2.2. Mesh topology

Mesh ingress is a feature that allows remote nodes to connect inbound to the control plane. This is especially useful when creating remote nodes in restricted networking environments that disallow inbound traffic.

Mesh ingress architecture showing the peering relationship between nodes.

Below are sample values used to configure each node in a mesh ingress topology:

Instance type

Hostname

Listener port

Peers from control nodes

Peers

Control plane

awx-task-65d6d96987-mgn9j

n/a

n/a

[hop node]

Hop node

awx-mesh-ingress-1

27199

True

[]

Execution node

awx-example.com

n/a

False

[hop node]

To create the mesh ingress, create an AWXMeshIngress resource in the namespace of the platform. The Koios operator installs its type and reconciles it. deployment_name is the controller of the platform (<platform>-controller); on OpenShift, ingress_type: route publishes it through a passthrough Route on port 443:

kubectl apply -n koios -f - <<'EOF'
apiVersion: awx.ansible.com/v1alpha1
kind: AWXMeshIngress
metadata:
  name: koios-mesh
spec:
  deployment_name: koios-controller
  ingress_type: route
EOF

The controller registers the mesh ingress as a hop node with the external address of the Route. To use it from a node:

  1. Make sure the node resolves the Route’s host name (kubectl get route -n koios koios-mesh -o jsonpath='{.spec.host}'). Nodes outside the cluster often use a public DNS that does not know the cluster’s domain.

  2. Edit the external instance and clear Peers from control nodes. On its Peers tab, use Associate peers to select the mesh ingress listener address, so the node connects inbound and no control node tries to reach it.

  3. Download the node’s install bundle again and run it (see Add an instance).

For other options, see Mesh Ingress in the upstream AWX Operator documentation. Then add the node to an instance group, as with any execution node. Whatever execution environment image is used to run a playbook must be pullable from the node (see Using a private image for the default EE), and everything the playbook uses must be reachable from it.

Job template using the instance group with the execution node to run jobs.

9.3. Add an instance

To register an external execution or hop node:

  1. Open Automation Execution > Infrastructure > Instances and click Create instance.

  2. Set Host name to the node’s resolvable hostname or IP address. Choose Execution or Hop in Instance type. These values cannot be changed after creation.

  3. Set Listener port when the node will accept inbound Receptor connections. Select Peers from control nodes only when control nodes should initiate connections to this listener; a listener port is required for that option.

  4. Under Options, select Enable instance to allow execution jobs and Managed by policy to allow automatic instance-group membership. Instance state is read-only and initially installed. Associate outbound peers after creating the instance.

Create instance form with hostname, type, listener port and options.
  1. Click Create instance. The Details tab opens. For existing instances, use Edit instance and Save instance for permitted changes. Managed control instances are maintained by the operator and cannot be deprovisioned through this workflow.

Details of an external execution instance registered but not installed yet, reported as Unavailable.
  1. If outbound peering is required, configure the instance’s Peers tab as described below before downloading its bundle.

  2. On Details, use the download control in Download bundle. The bundle contains private node keys as well as certificates, Receptor configuration and the installation playbook. Keep it private and do not attach it to issues or documentation.

Instance Details showing the Download bundle field and download control.
  1. Extract the bundle in a private working directory on a separate administration machine with access to the external node. Install its declared collection dependencies there:

ansible-galaxy collection install -r requirements.yml
  1. Set ansible_user and ansible_ssh_private_key_file in the bundle’s inventory.yml for the account that provisions the remote node. The supplied inventory already sets ansible_host to the registered hostname. These SSH credentials are for provisioning the external machine; they are not credentials for logging in to the platform.

  2. Run the bundle from that administration environment:

ansible-playbook -i inventory.yml install_receptor.yml

The playbook uses privilege escalation on the remote node. It creates the Receptor account and configures Receptor; execution nodes also need Podman and Ansible Runner. Review the downloaded requirements.yml and playbook before applying them. See Preparing the node for the documented node preparation requirements.

Note

Running a bundle again does not replace the node’s certificates if they already exist. When moving a node to another controller or rotating its certificates, download the appropriate bundle and use -e receptor_replace_tls=true. Otherwise, the existing certificates may still be signed by the previous controller’s CA and fail validation.

For an execution node, use Run health check and wait for the asynchronous result. Verify that the instance is ready, enabled and assigned to the instance group your job template uses before launching a job there. A hop node routes traffic and does not provide execution capacity.

9.3.1. Outbound peers

Open the instance’s Peers tab to inspect its outbound peer addresses.

Peers tab showing the listener addresses used for outbound connections.

Click Associate peers, select eligible listener addresses and confirm the selection. To remove associations, select rows and use Disassociate peers. Each association selects a Receptor listener address.

The Select peer addresses window; empty when no other instance has a listener address.

The controller rejects self-peering, duplicate peers on one instance and a direct reverse peer relationship. Select the direction that matches network reachability. Hop-to-hop connections are not categorically prohibited. Peers from control nodes controls automatic inbound peering from control nodes and is separate from this outbound list.

After changing a remote node’s peers or listener configuration, download its updated bundle and run it again to apply the configuration on that machine. Changing the controller record alone does not reconfigure remote Receptor. For a graphical view, see Topology Viewer.

9.4. Manage instances

The Instances list under Automation Execution > Infrastructure > Instances supports filtering, sorting and selecting instances for actions.

Instances list with control and execution nodes and their available actions.

Open an instance’s Details to inspect Node type, Status, Instance groups, Used capacity, Running jobs, Memory, Last health check, Forks, Enabled and any Errors. Fields vary by node type. The Forks control adjusts capacity_adjustment; it is not a live CPU utilization limit. See Improving Automation Controller Performance.

9.4.1. Instance states

  • installed: the instance is registered but has not yet completed a successful health check. Registration does not prove that the bundle has been run.

  • ready: the instance has reported usable health data. An execution instance must also be enabled and have capacity for a job to run.

  • unavailable: the instance is offline or its health check reported a problem; inspect Errors.

  • deprovisioning: removal has been requested and the controller is draining work before deleting the record.

The model also has provisioning and failure states for lifecycle operations. They do not mean this UI automatically provisions a newly registered remote machine. While an execution-node health check is pending, the UI shows a running status indicator.

9.4.2. Remove an external instance

Disable the instance first if you want to stop assigning new jobs while reviewing removal. Select the external instance and use Remove instance, review the confirmation, then click Remove instance to request deprovisioning. The controller waits for its running or waiting jobs before deleting the instance record.

Permanently remove instances confirmation for an external execution node.

Removal does not uninstall software or erase keys on the external machine. Review the remaining topology and rerun affected remote nodes’ bundles when peer configuration changes. Do not remove or recreate managed control instances to scale the controller; use the operator.

9.4.3. Execution-node health checks

Use Run health check from the execution instance’s row or details actions. The API queues the check asynchronously; accepting the request does not mean the node passed. Checks are supported for execution nodes, not hop nodes.

Execution instance Details after a failed health check, with the error in Errors.

Refresh the page to inspect Last health check, Status and Errors after the task finishes. A pending check temporarily disables another check request.

Execution instance Details with a health check still running.

For connectivity errors, inspect DNS, the chosen listener and peer direction, certificate trust and the external node’s Receptor service. The topology view can help locate the affected connection:

Topology view with an instance that reports a connection or health error.

9.5. Using a custom Receptor CA

Set the fields described in Custom Receptor CA (upstream AWX Operator documentation) under spec.controller of the KoiosPlatform.

9.6. Using a private image for the default EE

An external node pulls the image of the execution environment with podman, using only the credential of the execution environment. For an image in a private registry, such as ghcr.io/koiosplatform/koios-ee:

  1. In Automation Execution > Infrastructure > Credentials, create a credential of type Container Registry, with the registry host as Authentication URL (for example ghcr.io) and a user and token that can only read images.

  2. In Automation Execution > Infrastructure > Execution Environments, set that credential on the execution environment the jobs of the node use. You can keep a separate execution environment for external nodes, with the same image and the credential, and leave the default one as it is.

The credential is also used inside the cluster: for jobs in a container group, the controller turns it into the pull secret of the job’s pod, instead of the one of the namespace. That is why a separate execution environment for external nodes keeps the default one unchanged. See Execution Environments.

9.7. Troubleshooting

If you encounter issues while setting up instances, refer to these troubleshooting tips.

9.7.1. Fact cache not working

Check that the job template has Enable fact storage selected and the playbook gathers the facts you expect. Use Output to distinguish a fact-gathering failure from an unreachable host. Fact-cache artifacts are handled by the execution worker and returned to the controller; do not try to fix controller cache issues by editing files on a cluster node.

9.7.2. Container permission errors

A failure such as exec container process ... Permission denied needs diagnosis in the execution environment and on the node that ran the job. Check image permissions, rootless Podman prerequisites and any SELinux denials on an external execution node. For a container group, inspect the job pod’s security context and the cluster’s admission policies.

Do not disable SELinux or add label=disable as a blanket workaround. DEFAULT_CONTAINER_RUN_OPTIONS applies to Podman execution and does not configure container group pod security. Use supported image, storage and security-context configuration for the execution path you are using. See Deployment-managed settings and Execution Environments.