Investigate a failed run

Goal

For platform teams. In this task you find a run that failed, you read which type of resource failed, and you decide what to do next. A run is one plan or one apply that the control plane carried out.

Before you start

You have the CLI, and you are signed in. Sign in and get access shows how.
Know that an operator sees every run, and a person with the role owner:<team> sees the runs of the applications of that team.
Keep Run results, refusal reasons and exit codes open. It explains every status word on this page.

Steps

Find the run

The feature team can give you the ID of the run. Without it, list the runs. The newest run is first. Each filter narrows the list:

Option	What it keeps
`--application=<id>`	Runs of one application.
`--environment=<name>`	Runs in one environment.
`--kind=<kind>`	Only `plan` or only `apply`.
`--status=<status>`	Only one status, for example `failed`.
`--limit=<n>`	At most this many runs, from 1 to 1000. The default is 20.

This lists the runs that failed:

joomiz runs --status failed

ID                                    KIND   APPLICATION                  ENVIRONMENT  STATUS  STARTED
fe345a3f-0711-420d-bfb2-8b0d80acdf27  apply  quick-start-shop (9a3e62e6)  test         failed  2026-10-28T19:13:09.750Z

A run that applied only a part of its changes has its own status:

joomiz runs --status partially_applied

ID                                    KIND   APPLICATION                  ENVIRONMENT  STATUS             STARTED
376cfb8f-7140-4780-b51c-d0ccfdf09c29  apply  quick-start-shop (9a3e62e6)  test         partially_applied  2026-10-28T19:13:23.362Z

The dashboard has the same list on the page Runs. It filters by application ID, environment, kind, status, and the time of the start.

Read the run

Read one run with its ID:

joomiz runs show fe345a3f-0711-420d-bfb2-8b0d80acdf27

run fe345a3f-0711-420d-bfb2-8b0d80acdf27
  kind:         apply
  application:  quick-start-shop (9a3e62e6-4698-444b-86c9-5a379203ba90)
  environment:  test
  caller:       owner:finance-team
  status:       failed
  started:      2026-10-28T19:13:09.750Z
  finished:     2026-10-28T19:13:09.761Z
  duration:     11ms
  revision:     60fd7f09-0e9c-4cc5-92c4-3b7063a0a142
  plan:         f12817ab-dd95-4563-82d7-ff945c5c810e
  apply:        ab36e9af-e5ff-4077-8ee4-735086ae466a
groups:
  managed-resource/postgres  failed  3ms  created 0, updated 1, removed 0

Read the output from the top:

status is the result of the whole run.
caller is the role that started the run. This documentation says acted as.
revision and plan name the file and the plan that the run used.
groups: has one line for each kind and type of resource of the run. The control plane calls such a line a group. Each line has its state, its time and the number of changes.

A group has one of these states: queued, running, succeeded, failed, skipped and accepted. Run results, refusal reasons and exit codes explains them.

Here the type managed-resource/postgres failed. The run shows which type failed and how many changes it had. It does not show the reason, and it does not name single resources. The dashboard page of a run shows the same facts: its table of types has the columns Kind, Type, State, Duration and Changes.

Read the log

The log of a run lists its events in order. joomiz runs watch prints the whole log of a run that has ended, and then it stops:

joomiz runs watch fe345a3f-0711-420d-bfb2-8b0d80acdf27

watching run fe345a3f-0711-420d-bfb2-8b0d80acdf27
#1  2026-10-28T19:13:09.750Z  run_started  status running
#2  2026-10-28T19:13:09.754Z  group_queued  managed-resource/postgres
#3  2026-10-28T19:13:09.755Z  group_running  managed-resource/postgres
#4  2026-10-28T19:13:09.758Z  group_finished  managed-resource/postgres  failed  3ms  created 0, updated 1, removed 0
#5  2026-10-28T19:13:09.761Z  run_finished  status failed
run fe345a3f-0711-420d-bfb2-8b0d80acdf27 finished with status failed

The log shows when the work of each type started and ended, with its outcome. A type that waits for another one starts later. A type that does not start, because a type that it needs did not finish, is skipped.

Find the reason

The reason of a failure is in the output that the feature team saw when it ran plan or apply. The run history does not keep it. This is what that output looked like for this run:

error: the control plane could not fully apply joomiz.yaml to environment test (via https://joomiz.example.com/apply?environment=test)
  managed-resource/postgres: failed — the provisioner refused the connection
    shop-db (updated): failed — the provisioner reported no result for this resource
apply failed (plan f12817ab-dd95-4563-82d7-ff945c5c810e, apply ab36e9af-e5ff-4077-8ee4-735086ae466a)

The texts after failed name the cause. The common ones are:

Text	What it means	What to do
`the provisioner refused the connection`	Nothing listens at the registered address.	Check that the provisioner runs, and check its address in the registry.
`the provisioner sent an unexpected response`	The provisioner answered in a way that the contract does not allow, or the control plane did not send the call.	Check the provisioner log and the setting `joomiz.provisioners.allowed-networks`.
`the provisioner did not answer in time`	The provisioner is slow, or the network drops the call.	Check the provisioner, and the network between it and the control plane.
`the provisioner's host could not be resolved`	The name in the registered address has no address.	Check the address in the registry, and your DNS.
`the provisioner reported no result for this resource`	The call failed or the answer had no result for the resource.	Read the line above it.
`not planned: a resource type it depends on could not be planned`	A type that this type uses failed first, or it works in the background.	Fix the first failure. For slow work, read Support slow work in a provisioner.
`not applied yet: the provisioner accepted these changes and is still making them`	The provisioner works in the background.	Follow the run with `joomiz runs watch`.

If the feature team has no output, ask it to run joomiz plan again for the same environment. A plan changes nothing. It shows what the control plane would do now, and it prints the current reason for each failure.

Read a partly applied run

The status partially_applied means that some changes were made and some were not. Here the database was updated, and the deployment failed:

run 376cfb8f-7140-4780-b51c-d0ccfdf09c29
  kind:         apply
  application:  quick-start-shop (9a3e62e6-4698-444b-86c9-5a379203ba90)
  environment:  test
  caller:       owner:finance-team
  status:       partially_applied
  started:      2026-10-28T19:13:23.362Z
  finished:     2026-10-28T19:13:23.383Z
  duration:     21ms
  revision:     026d73b9-b9c7-4118-8c34-7d1e297afb0a
  plan:         430d0e9b-1a59-4f18-a9d5-fd3dddcf7651
  apply:        2baeea46-77cf-4302-b59d-2ea660d3a2fd
groups:
  managed-resource/postgres     succeeded  13ms  created 0, updated 1, removed 0
  deployment/cloud-foundry-app  failed     2ms  created 0, updated 1, removed 0

watching run 376cfb8f-7140-4780-b51c-d0ccfdf09c29
#1  2026-10-28T19:13:23.362Z  run_started  status running
#2  2026-10-28T19:13:23.365Z  group_queued  managed-resource/postgres
#3  2026-10-28T19:13:23.365Z  group_queued  deployment/cloud-foundry-app
#4  2026-10-28T19:13:23.366Z  group_running  managed-resource/postgres
#5  2026-10-28T19:13:23.379Z  group_finished  managed-resource/postgres  succeeded  13ms  created 0, updated 1, removed 0
#6  2026-10-28T19:13:23.380Z  group_running  deployment/cloud-foundry-app
#7  2026-10-28T19:13:23.382Z  group_finished  deployment/cloud-foundry-app  failed  2ms  created 0, updated 1, removed 0
#8  2026-10-28T19:13:23.383Z  run_finished  status partially_applied
run 376cfb8f-7140-4780-b51c-d0ccfdf09c29 finished with status partially_applied

The environment is now part way through the plan. Do this:

Find the type with the state failed, and fix its cause. Start the provisioner, or correct its settings.
Ask the feature team to run joomiz apply again with the same file. It is safe to repeat, because every provisioner must give the same end state when it repeats a change.
If someone else applied to the environment in the meantime, the control plane refuses the old plan as stale. Then the team runs joomiz plan and joomiz apply again.

Use the trace ID

When your control plane sends traces, joomiz runs show prints a line trace. Search for that ID in your tracing tool. You see every call that the run made to a provisioner. Check health and follow a run shows what the traces hold.

run 7bb5b27f-a1f9-49ae-9a5b-daaa715aedcf
  kind:         apply
  application:  quick-start-shop (75c7e55d-aaa1-48ab-bc58-d7ca3b44b68c)
  environment:  test
  caller:       owner:finance-team
  status:       succeeded
  started:      2026-10-28T19:22:00.755Z
  finished:     2026-10-28T19:22:00.791Z
  duration:     36ms
  revision:     defd57f2-d868-4b99-81ae-f74c99fc1fa1
  plan:         66a4d0e7-7191-4f6a-96ee-cf3f0277a99c
  apply:        0a6dc56b-2f75-4936-bb99-a0d52edeb311
  trace:        8f67f129ad08209b1f8df29d75874a05
groups:
  managed-resource/postgres     succeeded  15ms  created 1, updated 0, removed 0
  target/cloud-foundry-space    succeeded  12ms  created 1, updated 0, removed 0
  deployment/cloud-foundry-app  succeeded  8ms  created 1, updated 0, removed 0

A run on a control plane that sends no traces has no trace line.

Check

The investigation is complete when you can answer these questions:

Which type of resource failed, and which state does it have?
Which text names the cause?
Does the environment hold a half-finished change?
Who must act next: the team of the provisioner, or the feature team?

After the fix, the next apply of the feature team ends with the status succeeded.

Troubleshooting

Symptom	Cause	Fix
`no runs found`	No run matches the filters, or you may not see the run.	Remove a filter. An owner sees only the runs of its own applications.
`404` and `Run not found`	The ID is wrong, or the run belongs to another team.	Check the ID.
The status is `refused` with a reason in brackets	The control plane did not start the run.	Look up the reason in Run results, refusal reasons and exit codes.
The status is `accepted` for a long time	A provisioner works in the background, and the longest wait has not passed.	Wait, or release the lock. Release a stuck apply lock shows how.
The status is `abandoned`	An operator released the lock, or the control plane found no event for a long time.	Read the line `ended`.
Exit code `4` of `runs show`	The run has no final status yet.	Follow it with `joomiz runs watch`.

Read who did what in Read and manage the audit trail.