Support slow work in a provisioner

Goal

For platform teams. In this task you make a provisioner accept work that takes minutes, for example a pipeline run. The provisioner answers at once and finishes later. The control plane asks for the result until the work ends.

Before you start

You have a provisioner that answers POST /plan and POST /apply. Build a provisioner with the starter shows how.
You know the longest time that your work can take. The control plane accepts at most 30 minutes (PT30M).
You have the role operator to register the provisioner.

Steps

Answer 202 with an operation ID

When your provisioner cannot finish a plan or apply call in a short time, it starts the work and answers 202. The body holds one field, operationId. You choose the ID, and it must be a UUID. Generate it with a random source. It identifies the work. This call sends the file plan-request.json of Build a provisioner with the starter to a provisioner that accepts slow work:

curl -s -w '\nHTTP %{http_code}\n' -X POST "$PROVISIONER_URL/apply" -H 'Content-Type: application/json' -d @plan-request.json

{
  "operationId": "b09a01c4-61de-4dab-ad8e-f5f90097b8f7"
}
HTTP 202

One operation covers the whole change set of the call. The control plane records the operation, ends the call for the resources of that type, and continues with the other types. The work may already be under way when you answer, so apply must give the same end state when it runs again.

A call that does not need to wait still answers 200 with the result. A provisioner can use both ways, a different one for each call.

Serve GET /operations/{operationId}

The control plane calls GET /operations/{operationId} on your provisioner. By default it does this every 10 seconds. The platform team sets the time with joomiz.jobs.poll-interval (see Settings reference). The answer is always 200, and the body has the same shape as the answer of plan and apply.

While the work runs, answer "outcome": "in_progress" and a short progress note. Leave results empty. The control plane reads results only from the final answer.
When the work ends, answer succeeded or failed, with one result for each change. A failed answer is still 200.
For an ID that your provisioner does not know, answer 404. The control plane then fails the work at once and does not ask again.

After a final answer, the control plane never asks about that ID again. Never answer in_progress after a final answer. A 5xx answer, a timeout or a refused connection says nothing about the work, so the control plane asks again later.

curl -s -w '\nHTTP %{http_code}\n' "$PROVISIONER_URL/operations/$OPERATION_ID"

{
  "outcome": "in_progress",
  "description": "Pipeline run 4821 is at the deploy stage.",
  "results": []
}
HTTP 200

A later call, when the work has ended, answers with the result:

{
  "outcome": "succeeded",
  "description": "Handled the database shop-db.",
  "results": [
    {
      "id": "shop-db",
      "environment": "test",
      "outcome": "succeeded",
      "description": "Handled the database shop-db.",
      "outputs": {
        "password": "https://vault.example.com/secrets/shop-db-password"
      }
    }
  ]
}
HTTP 200

Register the longest wait

Register the provisioner with maxOperationDuration, an ISO 8601 duration of at most PT30M. Without it, the control plane treats an answer of 202 as an error and fails the work:

error: the control plane could not plan joomiz.yaml in environment test (computed by https://joomiz.example.com/plan?environment=test)
  target/cloud-foundry-space: succeeded — Planned 1 change for target/cloud-foundry-space.
    test-foundation (created): succeeded — Would create 'test-foundation' (target/cloud-foundry-space) in environment 'test'. Nothing was changed.
      outputs.org-guid: 3b1f6c0e-52a4-4d8e-9a7b-0c4e8d2f7a11
      outputs.space-guid: 9d2e4b7a-61c3-4f05-8b3a-5e7c1a0d9f22
  managed-resource/postgres: failed — not planned: the provisioner accepted this work asynchronously without being registered to, so there is no declared duration to bound it by
    shop-db (created): failed — not planned: the provisioner accepted this work asynchronously without being registered to, so there is no declared duration to bound it by
  deployment/cloud-foundry-app: failed — not planned: a resource type it depends on could not be planned
    shop-test-deployment (created): failed — not planned: a resource type it depends on could not be planned
plan failed (revision 7d4ac5d4-7e69-44d5-99d7-e003e5e71152, plan 50fe8a29-82ee-4766-b42c-eaa6d5d6c809)

Register a provisioner shows the registration. This is a registration with a longest wait of 5 minutes. The field maxOperationDuration holds it:

curl -s -w '\nHTTP %{http_code}\n' --variable '%JOOMIZ_TOKEN' --expand-header 'Authorization: Bearer {{JOOMIZ_TOKEN}}' -X POST "$JOOMIZ_SERVER_URL/provisioners" -H 'Content-Type: application/json' -d '{"name": "splunk-provisioner", "url": "https://splunk-provisioner.example.com", "types": [{"kind": "consumed-resource", "type": "splunk-index"}], "maxOperationDuration": "PT5M"}'

{"id":"65fab26b-5a25-4b5d-baed-034d455e608e","name":"splunk-provisioner","url":"https://splunk-provisioner.example.com","types":[{"kind":"consumed-resource","type":"splunk-index"}],"createdAt":"2026-10-28T18:28:19.980Z","updatedAt":"2026-10-28T18:28:19.980Z","maxOperationDuration":"PT5M"}
HTTP 201

The control plane stops waiting when the longest wait has passed since it accepted the operation. It then ends the work as failed.

Use the starter

The starter library serves GET /operations/{operationId} for you. You supply one bean that implements the interface ProvisionerOperations. The starter then uses it for every plan and apply call. Without this bean, the provisioner answers 200 to every call and 404 to every status check.

The interface has two methods:

interface ProvisionerOperations {
    fun accept(kind: ProvisionerOperationKind, changeSet: ResourceChangeSet): UUID?
    fun read(operationId: UUID): OperationOutcome?
}

enum class ProvisionerOperationKind { PLAN, APPLY }

accept runs for each POST /plan and POST /apply call. The value kind says which of the two it is. To take the work for later, start it and return an operation ID. The starter then answers 202. To answer at once, return null. The call then goes to your ProvisionerBehavior and answers 200.
read runs for each GET /operations/{operationId}. Return OperationOutcome.inProgress(description) while the work runs. Return OperationOutcome.finished(outcome) when it ends, with the ProvisionerOutcome that the call would have answered. Return null for an ID that you do not know. The starter then answers 404.
A OperationOutcome that is in progress holds no results.

This is a skeleton. It runs the work of your behavior in the background:

import com.joomiz.provisioner.api.OperationOutcome
import com.joomiz.provisioner.api.Outcome
import com.joomiz.provisioner.api.ProvisionerOutcome
import com.joomiz.provisioner.api.ResourceChangeSet
import com.joomiz.provisioner.behavior.ProvisionerBehavior
import com.joomiz.provisioner.operations.ProvisionerOperationKind
import com.joomiz.provisioner.operations.ProvisionerOperations
import org.springframework.stereotype.Component
import java.util.UUID
import java.util.concurrent.CompletableFuture
import java.util.concurrent.ConcurrentHashMap

@Component
class BackgroundOperations(private val behavior: ProvisionerBehavior) : ProvisionerOperations {

    private val operations = ConcurrentHashMap<UUID, CompletableFuture<ProvisionerOutcome>>()

    override fun accept(kind: ProvisionerOperationKind, changeSet: ResourceChangeSet): UUID? {
        if (operations.size >= MAX_OPERATIONS) return null
        val id = UUID.randomUUID()
        operations[id] = CompletableFuture.supplyAsync {
            when (kind) {
                ProvisionerOperationKind.PLAN -> behavior.plan(changeSet)
                ProvisionerOperationKind.APPLY -> behavior.apply(changeSet)
            }
        }
        return id
    }

    override fun read(operationId: UUID): OperationOutcome? {
        val operation = operations[operationId] ?: return null
        if (!operation.isDone) return OperationOutcome.inProgress("The work is still running.")
        operations.remove(operationId)
        val outcome = runCatching { operation.join() }
            .getOrElse { ProvisionerOutcome(Outcome.FAILED, "The work failed.", emptyList()) }
        return OperationOutcome.finished(outcome)
    }

    companion object {
        const val MAX_OPERATIONS = 100
    }
}

Follow these rules in your own class:

Generate the ID with a random source, as UUID.randomUUID() does. The ID must be a UUID.
Make read safe to repeat. It must not start work or change the operation.
After you answer succeeded or failed, the control plane never asks again about that ID. So you may forget the operation.
Keep an operation until its work ends. A provisioner that restarts and loses its operations answers 404, and the control plane then fails the work.
Hold a fixed number of operations at most, as MAX_OPERATIONS does. When you hold that many, return null. The call then runs at once.
Forget an operation when the longest wait that you registered has passed. The control plane no longer asks about it.

The starter also has a stub that simulates slow work. The settings provisioner.stub.async.enabled and provisioner.stub.async.in-progress-polls make it answer 202, and then in_progress for a number of polls. Use it to test the registration of the longest wait.

Check

Run a plan with a type that your provisioner handles. The CLI ends, and the run is waiting for provisioner. Its status is accepted:

error: the control plane could not plan joomiz.yaml in environment test (computed by https://joomiz.example.com/plan?environment=test)
  consumed-resource/splunk-index: pending — not planned yet: the provisioner accepted this work and is still doing it, as operation 80497595-0f44-47fb-8748-187c0d6eea29
    shop-logs (created): failed — not planned yet: the provisioner accepted this work and is still doing it, as operation 80497595-0f44-47fb-8748-187c0d6eea29
  managed-resource/azure-key-vault: succeeded — Planned 1 change for managed-resource/azure-key-vault.
    shop-vault (created): succeeded — Would create 'shop-vault' (managed-resource/azure-key-vault) in environment 'test'. Nothing was changed.
  target/kubernetes-namespace: succeeded — Planned 1 change for target/kubernetes-namespace.
    test-cluster (created): succeeded — Would create 'test-cluster' (target/kubernetes-namespace) in environment 'test'. Nothing was changed.
  deployment/kubernetes-deployment: succeeded — Planned 1 change for deployment/kubernetes-deployment.
    shop-test-deployment (created): succeeded — Would create 'shop-test-deployment' (deployment/kubernetes-deployment) in environment 'test'. Nothing was changed.
plan failed (revision cdf3849a-7921-4622-b1f4-27957e37c613, plan 36bbe33b-9dc4-41d2-ac51-3ef4068535ff)

The CLI prints plan failed and exits with code 1 at this point, because the work for one type has not finished. That type is pending, and the text names the operation. The work is not lost.

List the runs, and follow the run until it ends:

ID                                    KIND  APPLICATION                ENVIRONMENT  STATUS    STARTED
dcd87845-89e9-4f5f-9c1c-cebc9374ee65  plan  container-shop (d542d194)  test         accepted  2026-10-28T17:42:35.656Z

watching run dcd87845-89e9-4f5f-9c1c-cebc9374ee65
#1  2026-10-28T17:42:35.656Z  run_started  status running
#2  2026-10-28T17:42:35.657Z  group_queued  consumed-resource/splunk-index
#3  2026-10-28T17:42:35.657Z  group_queued  managed-resource/azure-key-vault
#4  2026-10-28T17:42:35.657Z  group_queued  target/kubernetes-namespace
#5  2026-10-28T17:42:35.657Z  group_queued  deployment/kubernetes-deployment
#6  2026-10-28T17:42:35.659Z  group_running  consumed-resource/splunk-index
#7  2026-10-28T17:42:35.659Z  group_running  managed-resource/azure-key-vault
#8  2026-10-28T17:42:35.659Z  group_running  target/kubernetes-namespace
#9  2026-10-28T17:42:35.698Z  group_finished  consumed-resource/splunk-index  accepted  39ms  created 1, updated 0, removed 0  operation 80497595-0f44-47fb-8748-187c0d6eea29
#10  2026-10-28T17:42:35.705Z  group_finished  managed-resource/azure-key-vault  succeeded  46ms  created 1, updated 0, removed 0
#11  2026-10-28T17:42:35.705Z  group_finished  target/kubernetes-namespace  succeeded  46ms  created 1, updated 0, removed 0
#12  2026-10-28T17:42:35.706Z  group_running  deployment/kubernetes-deployment
#13  2026-10-28T17:42:35.743Z  group_finished  deployment/kubernetes-deployment  succeeded  37ms  created 1, updated 0, removed 0
#14  2026-10-28T17:42:35.746Z  run_finished  status accepted
the call has ended, and a provisioner is still working on one or more groups. Watching until they finish.
no new event for 30 s (last event #14, run still accepted)
#15  2026-10-28T17:43:09.731Z  group_resolved  consumed-resource/splunk-index  succeeded  34.0s  operation 80497595-0f44-47fb-8748-187c0d6eea29
#16  2026-10-28T17:43:09.731Z  run_resolved  status succeeded
run dcd87845-89e9-4f5f-9c1c-cebc9374ee65 finished with status succeeded

The events group_resolved and run_resolved show that the control plane read the final answer of your operation. The time of group_resolved is the real time of the work, with a delay of up to one poll interval (10 seconds by default).

When the run has ended, the same run shows the final result of each type:

run dcd87845-89e9-4f5f-9c1c-cebc9374ee65
  kind:         plan
  application:  container-shop (d542d194-4247-40fb-bef5-43fa29236c86)
  environment:  test
  caller:       owner:finance-team
  status:       succeeded
  started:      2026-10-28T17:42:35.656Z
  finished:     2026-10-28T17:43:09.731Z
  duration:     34.0s
  revision:     cdf3849a-7921-4622-b1f4-27957e37c613
  plan:         36bbe33b-9dc4-41d2-ac51-3ef4068535ff
groups:
  consumed-resource/splunk-index    succeeded  39ms  created 1, updated 0, removed 0  resolved after 34.0s  operation 80497595-0f44-47fb-8748-187c0d6eea29
  managed-resource/azure-key-vault  succeeded  46ms  created 1, updated 0, removed 0
  target/kubernetes-namespace       succeeded  46ms  created 1, updated 0, removed 0
  deployment/kubernetes-deployment  succeeded  37ms  created 1, updated 0, removed 0

The line caller says which role started the run. This documentation says acted as.

An apply follows the same path. The apply run is accepted until your operation ends, and then the run gets its final status. While the run is accepted, the CLI prints apply partially_applied and exits with code 1. Do not apply again. Follow the run with joomiz runs watch. While the run is accepted, it keeps the lock for the application and the environment. A person with the role operator can release it with joomiz runs abandon.

Troubleshooting

Symptom	Cause	Fix
`not planned: the provisioner accepted this work asynchronously without being registered to`	The provisioner answered `202`, but the registration has no `maxOperationDuration`.	Register the longest wait.
`400` for `maxOperationDuration` at registration	The duration is zero, is not valid, or is over `PT30M`.	Write a duration such as `PT10M`.
The run stays `accepted` and then fails	The work did not end within the longest wait.	Make the work faster, or register a longer wait up to `PT30M`.
The work fails at once after the first poll	The provisioner answered `404` for the operation ID, for example because it restarted and lost the work.	Keep a record of each operation until you give a final answer.
A type that uses the output of the slow type fails with `not planned: a resource type it depends on could not be planned`	The control plane cannot plan or apply a type that depends on a type that works in the background.	Use a provisioner that answers at once for a type that other types use.
The work fails with `the provisioner sent an unexpected response`	The answer of `GET /operations/{operationId}` does not follow the contract.	Compare it with Provisioner API reference.

Make sure that the control plane may call your provisioner in Allow the address of your provisioner.