Status and conditions

The status of a Release records what convoy did and what it sees on the cluster. The controller-manager and two loops of the observer write it. Each writer owns its own fields, so one writer never overwrites the fields of another writer.

Who writes the status#

Each writer uses a server-side apply to /status with its own field manager. conditions is a map with type as the key, so each writer owns only its own conditions.

Writer	Field manager	Conditions	Other fields
controller-manager	`convoy-release`	`Applied`, `RolloutSucceeded`, `LifecycleSucceeded`	`lastAttempted`, `lastSuccessful`, `lastHealthy`, `rollout`, `pin`, `waves`, `lifecycle`, `resolvedRetentionPolicy`, `reconciledAt`, `lastHandledSyncAt`, `lastHandledForceSyncAt`, `observedGeneration`
observer, drift loop	`convoy-observe`	`InSync`	none
observer, health loop	`convoy-health`	`Healthy`	`healthSummary`

lastAttempted and lastSuccessful#

status.lastAttempted records the most recent reconcile. The controller-manager writes it at each reconcile, also when the reconcile waits.

attemptedAt is the time of the reconcile.
resolvedSource, resolvedCluster, resolvedMutations and renderRef contain the values that the reconcile found before it stopped.
phase is Succeeded when convoy applied a render, and Failed when a step failed. When the reconcile waits, phase is empty. The API also defines Rendering and Syncing, but convoy does not set them.
failure.stage tells which step failed. See the table below.

status.lastSuccessful is a copy of lastAttempted from the most recent reconcile that applied a render. A reconcile that waits or fails does not change it. After a rollback, it contains the last healthy render, because that render runs on the cluster.

When lastAttempted.renderRef and lastSuccessful.renderRef are different, convoy did not apply the newest render yet. lastAttempted tells why.

failure.stage has these values:

`failure.stage`	Step
`SourceResolution`	Fetch of the source
`ClusterResolution`	Look-up of the target `Cluster`
`Rendering`	Helm values, rendering, mutations, and the load of a pinned render
`Syncing`	Dry run, store write, apply, sync waves, cluster waves and prune
`HookFailed`	A lifecycle hook with `onFailure: Fail`. `hookName` and `hookPhase` name the hook.

The API also defines MutationResolution. convoy does not set it. A mutation failure has the stage Rendering.

Where a reconcile can stop

Step	`failure.stage`	`Applied` reason	Retried with backoff
Fetch the source: error	`SourceResolution`	SourceNotReady	yes
Fetch the source: not supported	`SourceResolution`	Unsupported	no
Find the cluster: missing or not reachable	`ClusterResolution`	ClusterNotReady	yes
Read Helm values	`Rendering`	ValuesNotReady	yes
Render or mutate: error	`Rendering`	DeployFailed	yes
Render or mutate: not supported	`Rendering`	Unsupported	no
Load the pinned render: not available	`Rendering`	Pinned	no
Load the pinned render: read or parse error	`Rendering`	DeployFailed	no
Manual sync mode: no `/sync` request	none	WaitingForSync	no
Cluster wave: wait	none	WaitingForClusterWave	no
Cluster wave: timeout	`Syncing`	ClusterWaveFailed	no
Dry run: rejected	`Syncing`	Unappliable	no
Write to the render store	`Syncing`	DeployFailed	yes
`preSync` hook: running	none	WaitingForHook	no
`preSync` hook: failed	`HookFailed`	HookFailed	no
Apply a wave: error	`Syncing`	DeployFailed	yes
Sync wave: wait for the previous wave	none	WaitingForWaveHealth	no
Sync wave: timeout	`Syncing`	WaveFailed or RolloutFailed	no
Prune	`Syncing`	PruneFailed	yes
Start `postSync` hooks: error	`Syncing`	DeployFailed	yes
All steps done	none	Applied	no

Yes: with spec.syncPolicy.retryBackoff.initial, convoy uses that backoff. Without it, the work queue uses its default rate limit.
No: convoy reconciles again after spec.syncPolicy.interval, or when the Release changes. A Progressing rollout is checked again at least every 10 seconds.
A sync-wave timeout with failurePolicy.rollback: OnWaveFailure can roll back. Then the reason is RolledBack. See Progressive rollout.

Conditions#

Condition	Writer	When it is present
`Applied`	controller-manager	Always, after the first reconcile.
`RolloutSucceeded`	controller-manager	When `spec.failurePolicy` is set. The reason is the rollout state.
`LifecycleSucceeded`	controller-manager	When the Release has lifecycle hooks.
`InSync`	observer	After the first drift check.
`Healthy`	observer	After the first health check.

Applied is True in three cases:

convoy applied the render.
convoy holds the last healthy render after a rollback.
The Release is in manual sync mode, and the render did not change.

A wait is False, for example WaitingForSync.

Healthy is True with reason NotConfigured when the Release has no spec.healthAssessment. It is Unknown with reason Pending while the current render is not applied and assessed.

Applied · written by convoy-controller-manager (field manager convoy-release)

Reason	Status	Meaning
`Applied`	True	convoy applied the current render. In manual sync mode, the current render is equal to the last applied render.
`SourceNotReady`	False	convoy could not fetch the source. convoy retries.
`ValuesNotReady`	False	convoy could not read a Helm values layer (a ConfigMap, a Secret or a GitRepository path). convoy retries.
`ClusterNotReady`	False	The `Cluster` does not exist or is not reachable. convoy retries.
`Unsupported`	False	The Release uses a feature that convoy does not implement, for example a Git `semver` ref, Helm `dependencyMode: resolve` or `jsonnet` content. This is a configuration error, so convoy does not retry with backoff.
`Unappliable`	False	The target cluster rejected the render in the pre-apply server-side dry run. convoy applies nothing. A forced `/sync` skips the dry run.
`DeployFailed`	False	Rendering, mutation or the apply failed. The message and `status.lastAttempted.failure.stage` tell which step failed. convoy retries.
`PruneFailed`	False	convoy applied the render, but could not delete an object that the render no longer contains. convoy retries.
`Pinned`	False	convoy cannot apply the pinned render, because no render store is configured or the store does not contain that render.
`WaitingForSync`	False	Manual sync mode. The current render differs from the last applied render. convoy waits for a `/sync` request.
`WaitingForWaveHealth`	False	A sync wave waits until the resources of the previous wave are healthy.
`WaveFailed`	False	A sync wave timed out while it waited for the previous wave. The Release has no `failurePolicy`.
`RolloutFailed`	False	A sync wave timed out while it waited for the previous wave, and the `failurePolicy` did not roll back.
`RolledBack`	True	The rollout failed and convoy applied the last healthy render again.
`WaitingForClusterWave`	False	The Release waits for the Releases of a lower cluster wave on the same `Cluster`.
`ClusterWaveFailed`	False	The Releases of a lower cluster wave did not become ready before the cluster-wave timeout.
`WaitingForHook`	False	convoy waits for the workloads of a lifecycle hook phase to complete.
`HookFailed`	False	A lifecycle hook with `onFailure: Fail` failed. convoy stops the sync.

See Release conditions for all reasons.

Why there is no Ready condition#

Each condition has one writer. A Ready condition needs one writer that reads the conditions of the other writers. It also hides which part failed. Thus convoy does not write Ready. You combine the conditions.

Combine the signals#

To check that a new render runs and works:

lastAttempted.renderRef is equal to lastSuccessful.renderRef, and Applied is True.
InSync is True. The cluster agrees with the desired render.
Healthy is True. A reason Pending means that the observer did not assess this render yet.
With spec.failurePolicy, RolloutSucceeded is True and status.rollout.renderRef is the new render.

Each condition has observedGeneration. When it is lower than metadata.generation, the condition is from an older spec. A pin, a /sync request or a change to mutations or lifecycle does not change metadata.generation.

Timestamps#

reconciledAt is the time of the last status write by the controller-manager. lastAttempted.attemptedAt changes at each reconcile, so in practice it is the time of the last reconcile.
lastHandledSyncAt is the time of the last /sync request that convoy applied in manual sync mode. convoy sets it only after a successful apply. In automatic mode, convoy does not set it.
lastHandledForceSyncAt is the time of the last /sync?force=true request that convoy handled. convoy sets it at the first reconcile after the request, also when that reconcile fails.

Printer columns#

kubectl get releases shows State (the rollout state), and the status of Applied, InSync and Healthy. See Release for all columns.