Health loop

The health loop is the second controller in convoy-observer. It runs next to the drift loop in the same process. For each Release, it reads the ReleaseInventory, reads each recorded object from the target cluster and evaluates the health rules. It writes the result to the ReleaseInventory and to the Healthy condition. It also polls the workloads of lifecycle hooks. It never renders. For the rules and the states, see Health assessment.

Queue and wakeups#

The loop is a HealthController on the shared harness in internal/controller/base. The queue key is the namespace/name of the Release. These events add a key:

The three ReleaseInventory checks compare only fields that the controller-manager owns. So the health writes of the loop do not wake the loop again. A /sync annotation and a status write also do not wake it.

One pass#

(*HealthController).syncHandler does these steps in this order:

1. It reads the Release from the cache. If the Release does not exist, it returns. The controller-manager deletes the ReleaseInventory.
2. Shard gate. If this replica does not own the shard of the target Cluster, it returns.
3. Deletion. If the Release has a deletion timestamp, it only polls hooks. The section “Deletion” below shows the steps.
4. Hooks. pollHooks polls the hook workloads. A polling error is logged, not returned.
5. Health. Without spec.healthAssessment, it writes Healthy=True with reason NotConfigured and an empty summary. Else it calls assess and writes the result.
6. Wake the controller. On an edge, it sends POST /sync.
7. Next pass. scheduleHealthRecheck requeues the key.

Assessment#

assess works on the inventory, not on the render. So an object gets an assessment as soon as the controller records it after the apply of its wave.

1. collectRules collects the rules. It lists the GlobalHealthRuleSet objects that match globalRuleSetSelector. Then it lists the HealthRuleSet objects in the namespace of the Release that match localRuleSetSelector. A missing selector matches nothing.
2. health.NewEvaluator compiles all rules. A compile error stops the assessment of the full Release.
3. It reads the ReleaseInventory from the cache. No inventory, or an empty one, gives an empty result.
4. It reads the target Cluster. A missing or unreachable cluster stops the assessment.
5. priorAndAnchors reads the previous state and the firstUnhealthy time of each entry. It reads them before this pass writes.
6. assessItem evaluates each entry.
7. It sets renderAdvanced when an entry has a renderRef that is different from its assessedRenderRef.
8. writeInventoryHealth writes the result of each entry.

If assess returns an error, the pass writes Healthy=Unknown with reason Unknown. It keeps the previous healthSummary.

Per-entry evaluation#

assessItem gives one state for each inventory entry:

health.DeriveState uses the result of the rules and the firstUnhealthy anchor:

• An evaluation error gives Unknown and keeps the anchor.
• A pass gives Healthy and clears the anchor.
• A failure sets the anchor to now if it is not set. Without healthAssessment.deadline, the state is Progressing. When the time since the anchor is equal to or more than the deadline, the state is Degraded. Else the state is Progressing.

Anchors#

The loop stores firstUnhealthy in the ReleaseInventory. So an observer restart does not reset the clock of a failing object. The loop does not reset the anchor when the render changes.

The anchor controls only the difference between Progressing and Degraded. The rollout deadline does not use it. The controller measures the rollout deadline from status.rollout.startedAt. See Rollout state machine.

Writes#

The loop writes two objects with Server-Side Apply:

• writeInventoryHealth sends each entry with its keys and state, message, firstUnhealthy and assessedRenderRef. The field manager is convoy-health-inventory. assessedRenderRef is the renderRef of the entry, which is the render that the controller applied for this object. An empty set writes nothing.
• writeHealth sends the Healthy condition and healthSummary. The field manager is convoy-health. It skips the write when nothing changed.

The condition depends on renderCurrent. This is true when every entry has the renderRef of status.lastAttempted.

The /health subresource does not run an assessment. HealthResult builds the response from the entries of the ReleaseInventory. See Status ownership.

Hook polling#

pollHooks reads the hooks list of the ReleaseInventory from the cache. It polls an entry when the entry has a workloadName and one of these is true:

• The entry has no finishedAt.
• createdAt is after finishedAt. This means that the controller armed the hook again.

For each polled entry, pollHookWorkload reads the Job or Pod from the target cluster:

The timeout counts from createdAt. writeHookStatus writes state, message and finishedAt under the field manager convoy-hook-status. It sends every entry that has a state, not only the polled entries. The controller reads these fields to advance the hook phase. See Lifecycle hooks.

Wake the controller#

The loop sends POST releases/<name>/sync when one of these edges occurs in this pass:

• Wave complete. frontierWaveCompleted is true when all entries of the highest wave in the inventory are Healthy, and one or more of them were not Healthy before this pass.
• Render advanced. The assessment reached a render that the controller applied after the previous pass.
• Hook batch complete. One or more hooks reached a terminal state.

These checks are edge triggers. A steady pass does not send a request. The /sync request sets the annotation convoy.kumodo.io/sync-requested-at, which wakes the controller. If the request fails, the pass returns an error and the work queue retries the key.

The controller-manager also watches the ReleaseInventory. A change of state or assessedRenderRef wakes the controller without a /sync. See Reconcile flow.

Re-check backoff#

scheduleHealthRecheck chooses the delay of the next pass from a list of pending targets:

• Each object that is not Healthy.
• Each hook in the state Running.
• The target render-pending when the condition is Pending.

The loop keeps an attempt count for each target. The count is in memory and is lost at a restart. Each pass increments the count of each pending target and clears the other targets. progressBackoff converts a count into a delay:

The next pass uses the shortest delay of all pending targets. So a new pending target brings the delay back to 2 seconds. syncPolicy.interval is the maximum delay. Without an interval, the maximum is 32 seconds. The loop adds a random jitter of up to 20%.

When no target is pending, the loop clears the counts. It then requeues after syncPolicy.interval. Without an interval, only events and the informer resync start the next pass.

Deletion#

When the Release has a deletion timestamp, a pass does these steps:

1. pollHooks polls the hook workloads, for example the preDelete and postDelete hooks.
2. If a hook finished, the loop sends POST /sync. The controller then continues the finalization.
3. scheduleHealthRecheck uses only the running hooks as pending targets.

The loop does not assess the objects and does not write Healthy. When all hooks have a finishedAt, pollHooks writes nothing. So the loop stops writing to the ReleaseInventory before the controller deletes it. See Deletion.