Operational runbooks

Start with Check platform health and Logs and diagnostics. Use an explicit cluster context and namespace for every Kubernetes/OpenShift command. Apply supported platform changes through KoiosPlatform and application changes through the UI/API; generated Deployments, StatefulSet templates and service registrations are not repair interfaces.

The endpoint/workload baseline was verified on 2026-11-18. The failure-recovery procedures below are guidance from the operator and service sources; they have not been validated as end-to-end failure drills on Koios. Plan disruptive storage and restore work with the responsible administrator.

Platform does not reach Successful

Stage

Procedure

Symptom

Successful never becomes True, Failure=True, or reconciliation repeatedly reports the same error.

Check

Read all condition reasons/messages, the selected component names and actual workload readiness. Check idle mode or a restore hold first. Inspect namespace events, unbound PVCs and image pull failures. Inspect gateway operator logs, then the affected child operator’s logs. Awaiting next reconciliation alone is not a failure.

Action

Correct the reported setting in the owning CR through the platform’s declarative configuration. Resolve registry access, unavailable storage, quota or external database/CA dependencies through their approved configuration workflows. If an operator itself is unavailable, repair its installation using the matching installer/OLM procedure. Do not bypass reconciliation by editing generated Deployments.

Verify

Wait for Successful=True and Failure=False, check expected replicas and the service endpoints, then perform a representative application operation. Do not require Running=False.

Use the KoiosPlatform reference for supported fields and Install and upgrade for operator installation. Preserve the original error before changing several settings at once.

Gateway returns 502 or 503 for a service

Stage

Procedure

Symptom

The gateway responds, but a controller, EDA or hub request fails with 502/503.

Check

Compare gateway ping/ with the authenticated gateway status/ and the affected service’s endpoint from the health table. Inspect gateway api and proxy logs and the child service’s readiness/events/logs. Check that the component is enabled, not idle or still being restored. A successful gateway ping does not establish downstream health.

Action

Correct the failing service’s supported CR configuration or its external dependency. Let the operators restore workloads and service registration. A restore can temporarily return 503 while a service registers. Do not create registrations or point the proxy at another platform by hand.

Verify

The service endpoint responds with its expected status/body and the gateway no longer reports the affected service failure. Verify a job, activation or content operation for that service, as appropriate.

A 401 on the protected gateway status endpoint without credentials is expected; it is a different problem from a proxy 502/503. Follow Backup and restore if registration is still being rebuilt after a restore.

Controller job remains pending

Stage

Procedure

Symptom

A job remains Pending instead of starting.

Check

Open Automation Execution > Jobs and inspect the job’s details, dependencies and assigned instance group. Check Automation Execution > Infrastructure > Instance Groups, Instances and Topology View for available capacity and node health. Inspect controller task logs; for container groups, also inspect execution pod scheduling, resource/quota and image pull events.

Action

Correct instance-group assignments or supported concurrency/capacity settings in the UI/API. Reduce competing work or adjust supported platform resources through the CR when capacity is insufficient. Resolve blocked project/inventory updates or execution-image access through their application objects. Do not force job database states or delete running job pods to manufacture capacity.

Verify

The affected job starts and its final result/output is appropriate. Check the following queued job as well, so one successful retry does not hide persistent capacity starvation.

Use Performance, Instances, Topology Viewer and Instance Groups for the canonical capacity model. The controller web Deployment’s replica count alone is not a measure of job execution capacity.

EDA activation cannot start its decision environment

Stage

Procedure

Symptom

An activation remains Starting or fails before rulebook output appears.

Check

Open its History instance and inspect output; check activation-worker logs and runtime pod events. Confirm the decision environment’s full image reference, registry reachability and pull policy. Distinguish image authorization failures from scheduling/quota failures.

Action

Correct the decision environment in Automation Decisions > Decision Environments. For a separate private image, create the EDA Container Registry credential under Automation Decisions > Infrastructure > Credentials and select it in the decision environment. For the release image, reconcile the installer’s registry-access configuration/pull secret through the documented setup. A controller registry credential does not configure EDA. After correction, enable/restart the activation as appropriate; disable repeated failing instances while investigating.

Verify

A new History instance starts with the intended image. Expect Running for a listening rulebook or Completed for a finite one with restart policy Never. Follow a test event into Rule Audit and verify any controller job separately.

See EDA image pull troubleshooting and Registry access. Never resolve a pull failure by printing registry Secrets or embedding a password in the image URL.

Hub collection import fails

Stage

Procedure

Symptom

Upload is accepted but import fails, or the collection cannot be installed after publishing.

Check

Open View import logs from Automation Content > Collection Approvals, or the namespace’s Imports view (My imports). Inspect the matching task and hub worker log, not only the CLI’s final lines. Determine whether failure occurred during import, approval or client installation.

Action

Fix the reported collection metadata/content and rebuild. For example, include requires_ansible in meta/runtime.yml. If a version already exists, publish a new version rather than overwriting different content. After a successful import to staging, an authorized approver moves it to published through the UI/API. Do not alter hub database records or artifact files manually.

Verify

Import succeeds, the intended version is in published, and a clean client can install it from the published distribution. Staging membership alone does not establish installability from published.

Follow Publish your first collection and Hub troubleshooting for the full procedure. The missing requires_ansible, staging/published distinction and successful installation were validated in that walkthrough; other failures require their own diagnosis.

Managed PostgreSQL storage is full

Stage

Procedure

Symptom

Database writes fail, service logs report insufficient disk space, or database/storage alerts fire.

Check

Read status.postgresStatefulSet, inspect that StatefulSet’s pod/events and its actual PVC. kubectl get pvc shows allocated capacity and binding, not free filesystem space. Correlate database logs with storage/filesystem monitoring; distinguish a full database volume from node ephemeral storage or a full backup/content volume.

Action

Limit new work through application controls while planning recovery. Ask the storage administrator to expand the existing PostgreSQL PVC declaratively through the Kubernetes storage API, only if its StorageClass/CSI driver supports expansion and the filesystem can grow. Keep the CR’s existing PostgreSQL storage-size/class values: the current operator rejects changes to the StatefulSet’s immutable claim template; changing the CR size is not an automatic PVC-expansion interface. If expansion is unavailable, plan recovery to adequately sized storage using the canonical backup/restore procedure and a verified recoverable backup.

Verify

Confirm PVC resize completion, filesystem space and database readiness, then repeat service checks and a controlled write operation. PVC capacity alone does not verify filesystem growth or recovered database writes. Check that reconciliation remains successful.

PVC expansion and full-volume recovery have not been validated here. Do not patch StatefulSet claim templates, delete database/WAL files, delete the PVC or run ad hoc SQL cleanup. Preserve the last verified backup. Use Backup and restore for recovery planning, including destination storage overrides. A restore from an older backup has a recovery point determined by that backup, not by the time of the failure.

After recovery, review Management Jobs for supported controller data-retention work and Metrics for monitoring. Retention jobs need a functioning database and are not a universal emergency remedy for an already full disk.

After restoring a backup

Stage

Procedure

Symptom

A restored platform is still becoming available, or needs acceptance before use.

Check

Follow the restore resource’s completion status and restored-component list. Check the destination’s hostname, database/Redis isolation and intended storage against the restore guide. A restored copy includes schedules, credentials and integration targets from the source.

Action

Let the restore operator finish its hold/restore/resume sequence; do not start child Deployments manually. Use the restore’s supported overrides for a distinct destination, especially separate empty external databases/connection Secrets. Review schedules, activations, notifications, webhooks and external targets through the UI/API before allowing the copy to act on production systems. Use an isolated recovery environment when those targets must not be contacted during startup.

Verify

The restore reports complete and expected components restored; Successful=True and Failure=False follow on the new platform. Repeat workload and endpoint checks. In an isolated, approved scope, verify login, credential usability, a controller job, EDA operation and hub content access for enabled components. Confirm the source platform was not changed.

Use Backup and restore as the canonical resource/field reference. Keep backup copies and their encryption material under the documented access controls. Review the next backup policy for the recovered platform; an available login page alone is not restore acceptance.