Health assessment

Health assessment tells you if the applied resources of a Release work. The observer reads each resource from the target cluster and evaluates it with CEL rules. It gives each resource a health state. Then it sets the Healthy condition of the Release from these states.

The health loop#

The observer runs the health loop. It is a second loop next to the drift loop, and it works on one Release at a time. In each pass, the loop does these steps:

1. It reads the resources that the controller recorded in the ReleaseInventory of the Release.
2. It reads each resource from the target cluster and gives it a health state.
3. It writes the state of each resource back to the ReleaseInventory.
4. It writes the Healthy condition and status.healthSummary on the Release.

The loop does not render the source again. The ReleaseInventory is the list of resources to assess. Thus, the observer assesses a resource as soon as the controller applies it.

The loop also polls the workloads of lifecycle hooks. It does this also when the Release has no spec.healthAssessment.

Select the rules#

Set spec.healthAssessment to select health rule sets by label:

spec:
  healthAssessment:
    globalRuleSetSelector:
      matchLabels:
        convoy.kumodo.io/default: "true"
    localRuleSetSelector:
      matchLabels:
        team: payments
    deadline: 1m

• globalRuleSetSelector selects GlobalHealthRuleSet objects.
• localRuleSetSelector selects HealthRuleSet objects in the namespace of the Release.
• A selector that is not set selects no rule set. An empty selector {} selects all rule sets.
• convoy uses the rules of all selected sets together. No set has priority.

A rule applies to a resource when its apiVersion and one of its resources match the resource. The name of the resource does not matter.

The state of a resource#

For each resource in the ReleaseInventory, the observer finds the rules that apply:

• No rule applies. The resource is Healthy. The observer does not read it from the cluster.
• All rules return true. The resource is Healthy.
• A rule returns false. The resource is Progressing or Degraded. The message comes from the first rule that fails.
• A rule cannot be evaluated. The resource is Unknown. A rule that does not return a bool is also an evaluation error.
• The resource cannot be read. The resource is Unknown. This includes a resource that is not on the cluster.

Progressing and Degraded#

When a resource fails a rule for the first time, the observer records the time in the firstUnhealthy field of the ReleaseInventory entry. The deadline starts at this time:

• Before spec.healthAssessment.deadline passes, the resource is Progressing.
• After the deadline passes, the resource is Degraded.
• Without a deadline, a failing resource stays Progressing.

When the resource passes again, the observer clears firstUnhealthy. An Unknown result keeps firstUnhealthy, so a short read error does not start the deadline again. The value is in the ReleaseInventory, so it survives a restart of the observer.

A new render does not clear firstUnhealthy. A resource that fails before and after a new render can be Degraded immediately.

assessedRenderRef#

Each ReleaseInventory entry has two render references:

• renderRef: the render that the controller applied the resource with. The controller writes it.
• assessedRenderRef: the render that the observer assessed. The observer copies it from renderRef when it writes the state.

When the two values are equal, the state is for that render. The controller uses this to find out if the current render is healthy. Examples are runAfter: Healthy on a postSync hook and status.lastHealthy.

The Healthy condition#

The observer sets the Healthy condition with its own field manager. The condition shows the worst state of the resources:

• Pending. The current render is status.lastAttempted.renderRef. Before all inventory entries have this render, the health of the previous render does not count. During a rollout that holds at a sync wave, Healthy can stay Pending.
• Cannot assess. Examples are an unreachable cluster, a rule that does not compile, and a selector that is not valid. The message gives the cause. status.healthSummary keeps its last value.
• NotConfigured. convoy then treats all waves and all runAfter: Healthy hooks as healthy.

healthSummary#

status.healthSummary counts the resources in each state:

status:
  healthSummary:
    total: 3
    healthy: 2
    progressing: 1
    degraded: 0
    unknown: 0

The suspended field is never set. When spec.healthAssessment is not set, all counts are 0.

Read the state of each resource#

status does not hold the state of each resource. To read it, use the /health subresource:

kubectl convoy release health podinfo

The observer returns the states from the ReleaseInventory. It does not evaluate the rules again. See Release health and CLI.

How health wakes the controller#

The controller does not wait for its poll interval to see a health change:

• The controller watches the ReleaseInventory. When the state or the assessedRenderRef of an entry changes, convoy reconciles the Release again.
• The observer sends a /sync request when the highest applied sync wave becomes fully healthy.
• The observer sends a /sync request when it assesses a new render for the first time.
• The observer sends a /sync request when a lifecycle hook completes.

While a resource is not Healthy, the observer checks the Release again after 2 seconds. The delay increases in steps to 32 seconds. spec.syncPolicy.interval limits the delay. When all resources are Healthy, the observer checks again at spec.syncPolicy.interval.