Operational runbooks
Start with Check platform health and Logs and diagnostics. Use an explicit
cluster context and namespace for every Kubernetes/OpenShift command. Apply supported platform
changes through KoiosPlatform and application changes through the UI/API; generated Deployments,
StatefulSet templates and service registrations are not repair interfaces.
The endpoint/workload baseline was verified on 2026-11-18. The failure-recovery procedures below are guidance from the operator and service sources; they have not been validated as end-to-end failure drills on Koios. Plan disruptive storage and restore work with the responsible administrator.
Platform does not reach Successful
Stage |
Procedure |
|---|---|
Symptom |
|
Check |
Read all condition reasons/messages, the selected component names and actual workload readiness. Check idle mode or a restore hold first. Inspect namespace events, unbound PVCs and image pull failures. Inspect gateway operator logs, then the affected child operator’s logs. Awaiting next reconciliation alone is not a failure. |
Action |
Correct the reported setting in the owning CR through the platform’s declarative configuration. Resolve registry access, unavailable storage, quota or external database/CA dependencies through their approved configuration workflows. If an operator itself is unavailable, repair its installation using the matching installer/OLM procedure. Do not bypass reconciliation by editing generated Deployments. |
Verify |
Wait for |
Use the KoiosPlatform reference for supported fields and Install and upgrade for operator installation. Preserve the original error before changing several settings at once.
Gateway returns 502 or 503 for a service
Stage |
Procedure |
|---|---|
Symptom |
The gateway responds, but a controller, EDA or hub request fails with 502/503. |
Check |
Compare gateway |
Action |
Correct the failing service’s supported CR configuration or its external dependency. Let the operators restore workloads and service registration. A restore can temporarily return 503 while a service registers. Do not create registrations or point the proxy at another platform by hand. |
Verify |
The service endpoint responds with its expected status/body and the gateway no longer reports the affected service failure. Verify a job, activation or content operation for that service, as appropriate. |
A 401 on the protected gateway status endpoint without credentials is expected; it is a different problem from a proxy 502/503. Follow Backup and restore if registration is still being rebuilt after a restore.
Controller job remains pending
Stage |
Procedure |
|---|---|
Symptom |
A job remains Pending instead of starting. |
Check |
Open Automation Execution > Jobs and inspect the job’s details, dependencies and assigned instance group. Check Automation Execution > Infrastructure > Instance Groups, Instances and Topology View for available capacity and node health. Inspect controller task logs; for container groups, also inspect execution pod scheduling, resource/quota and image pull events. |
Action |
Correct instance-group assignments or supported concurrency/capacity settings in the UI/API. Reduce competing work or adjust supported platform resources through the CR when capacity is insufficient. Resolve blocked project/inventory updates or execution-image access through their application objects. Do not force job database states or delete running job pods to manufacture capacity. |
Verify |
The affected job starts and its final result/output is appropriate. Check the following queued job as well, so one successful retry does not hide persistent capacity starvation. |
Use Performance, Instances, Topology Viewer and Instance Groups for the canonical capacity model. The controller web Deployment’s replica count alone is not a measure of job execution capacity.
EDA activation cannot start its decision environment
Stage |
Procedure |
|---|---|
Symptom |
An activation remains Starting or fails before rulebook output appears. |
Check |
Open its History instance and inspect output; check activation-worker logs and runtime pod events. Confirm the decision environment’s full image reference, registry reachability and pull policy. Distinguish image authorization failures from scheduling/quota failures. |
Action |
Correct the decision environment in Automation Decisions > Decision Environments. For a separate private image, create the EDA Container Registry credential under Automation Decisions > Infrastructure > Credentials and select it in the decision environment. For the release image, reconcile the installer’s registry-access configuration/pull secret through the documented setup. A controller registry credential does not configure EDA. After correction, enable/restart the activation as appropriate; disable repeated failing instances while investigating. |
Verify |
A new History instance starts with the intended image. Expect Running for a listening rulebook or Completed for a finite one with restart policy Never. Follow a test event into Rule Audit and verify any controller job separately. |
See EDA image pull troubleshooting and Registry access. Never resolve a pull failure by printing registry Secrets or embedding a password in the image URL.
Hub collection import fails
Stage |
Procedure |
|---|---|
Symptom |
Upload is accepted but import fails, or the collection cannot be installed after publishing. |
Check |
Open View import logs from Automation Content > Collection Approvals, or the namespace’s Imports view (My imports). Inspect the matching task and hub worker log, not only the CLI’s final lines. Determine whether failure occurred during import, approval or client installation. |
Action |
Fix the reported collection metadata/content and rebuild. For example, include |
Verify |
Import succeeds, the intended version is in published, and a clean client can install it from the published distribution. Staging membership alone does not establish installability from published. |
Follow Publish your first collection and
Hub troubleshooting for the full procedure. The missing
requires_ansible, staging/published distinction and successful installation were validated in
that walkthrough; other failures require their own diagnosis.
Managed PostgreSQL storage is full
Stage |
Procedure |
|---|---|
Symptom |
Database writes fail, service logs report insufficient disk space, or database/storage alerts fire. |
Check |
Read |
Action |
Limit new work through application controls while planning recovery. Ask the storage administrator to expand the existing PostgreSQL PVC declaratively through the Kubernetes storage API, only if its StorageClass/CSI driver supports expansion and the filesystem can grow. Keep the CR’s existing PostgreSQL storage-size/class values: the current operator rejects changes to the StatefulSet’s immutable claim template; changing the CR size is not an automatic PVC-expansion interface. If expansion is unavailable, plan recovery to adequately sized storage using the canonical backup/restore procedure and a verified recoverable backup. |
Verify |
Confirm PVC resize completion, filesystem space and database readiness, then repeat service checks and a controlled write operation. PVC capacity alone does not verify filesystem growth or recovered database writes. Check that reconciliation remains successful. |
PVC expansion and full-volume recovery have not been validated here. Do not patch StatefulSet claim templates, delete database/WAL files, delete the PVC or run ad hoc SQL cleanup. Preserve the last verified backup. Use Backup and restore for recovery planning, including destination storage overrides. A restore from an older backup has a recovery point determined by that backup, not by the time of the failure.
After recovery, review Management Jobs for supported controller data-retention work and Metrics for monitoring. Retention jobs need a functioning database and are not a universal emergency remedy for an already full disk.
After restoring a backup
Stage |
Procedure |
|---|---|
Symptom |
A restored platform is still becoming available, or needs acceptance before use. |
Check |
Follow the restore resource’s completion status and restored-component list. Check the destination’s hostname, database/Redis isolation and intended storage against the restore guide. A restored copy includes schedules, credentials and integration targets from the source. |
Action |
Let the restore operator finish its hold/restore/resume sequence; do not start child Deployments manually. Use the restore’s supported overrides for a distinct destination, especially separate empty external databases/connection Secrets. Review schedules, activations, notifications, webhooks and external targets through the UI/API before allowing the copy to act on production systems. Use an isolated recovery environment when those targets must not be contacted during startup. |
Verify |
The restore reports complete and expected components restored; |
Use Backup and restore as the canonical resource/field reference. Keep backup copies and their encryption material under the documented access controls. Review the next backup policy for the recovered platform; an available login page alone is not restore acceptance.