Add health rules to a Release

Health rules tell convoy when a resource works. Without them, the Healthy condition of a Release is always True with reason NotConfigured. In this guide, you apply the default rule set for built-in workloads. You add a rule set for a custom resource. You select both in a Release with spec.healthAssessment. Then you read the result. For the full model, see Health assessment.

Before you start#

convoy runs on your cluster.
The Cluster dev-local and the GitRepository podinfo exist in the namespace default. See Deploy from Git.
You can create cluster-scoped objects. A GlobalHealthRuleSet is cluster-scoped.
The kubectl convoy plugin is on your PATH.

Steps#

1. Apply the default rule set#

The default rule set has rules for Deployments, StatefulSets, DaemonSets, Pods, Jobs, PersistentVolumeClaims and PersistentVolumes. It has the label convoy.kumodo.io/default: "true".

health/convoy-defaults.yaml
# Default health rules shipped out-of-box. A Release opts into them by setting
# spec.healthAssessment.globalRuleSetSelector to match the label below. Expressions
# use has() guards so objects mid-creation (no status yet) evaluate cleanly.
apiVersion: extensions.convoy.kumodo.io/v1alpha1
kind: GlobalHealthRuleSet
metadata:
  name: convoy-defaults
  labels:
    convoy.kumodo.io/default: "true"
spec:
  rules:
    - apiVersion: apps/v1
      resources: [deployments]
      # Healthy only when the ROLLOUT is complete, not merely "min replicas available".
      # availableReplicas alone can't tell an OLD available pod from a NEW one, so during
      # a rolling update's surge window (old pod still available, observedGeneration
      # already bumped) a bad render would read as healthy and the rollout would wrongly
      # succeed. Require: the new ReplicaSet fully scaled (updatedReplicas == spec), no
      # old/surge replicas remaining (replicas == updatedReplicas), and all of the NEW
      # replicas available — the kubectl-rollout-status condition.
      ruleExpression: >-
        has(object.status) && has(object.status.observedGeneration) &&
        object.status.observedGeneration == object.metadata.generation &&
        (object.spec.replicas == 0 ||
         (has(object.status.updatedReplicas) && object.status.updatedReplicas == object.spec.replicas &&
          has(object.status.replicas) && object.status.replicas == object.status.updatedReplicas &&
          has(object.status.availableReplicas) && object.status.availableReplicas == object.status.updatedReplicas))
      messageExpression: >-
        (has(object.status.availableReplicas) ? string(object.status.availableReplicas) : '0') +
        '/' + string(object.spec.replicas) + ' available, ' +
        (has(object.status.updatedReplicas) ? string(object.status.updatedReplicas) : '0') + ' updated'
    - apiVersion: apps/v1
      resources: [statefulsets]
      # As with Deployments, require the rollout complete: all replicas ready AND all at
      # the updated revision (updatedReplicas == spec), so a partially-rolled bad update
      # doesn't read healthy off the still-running old pods.
      ruleExpression: >-
        has(object.status) && has(object.status.observedGeneration) &&
        object.status.observedGeneration == object.metadata.generation &&
        (object.spec.replicas == 0 ||
         (has(object.status.readyReplicas) && object.status.readyReplicas == object.spec.replicas &&
          has(object.status.updatedReplicas) && object.status.updatedReplicas == object.spec.replicas))
      messageExpression: >-
        (has(object.status.readyReplicas) ? string(object.status.readyReplicas) : '0') +
        '/' + string(object.spec.replicas) + ' ready, ' +
        (has(object.status.updatedReplicas) ? string(object.status.updatedReplicas) : '0') + ' updated'
    - apiVersion: apps/v1
      resources: [daemonsets]
      # Require the rollout complete: every desired pod ready AND updated to the new
      # pod template (updatedNumberScheduled == desiredNumberScheduled), so a rolling
      # DaemonSet update to a bad image doesn't read healthy off the not-yet-updated nodes.
      ruleExpression: >-
        has(object.status) &&
        object.status.observedGeneration == object.metadata.generation &&
        has(object.status.numberReady) && has(object.status.desiredNumberScheduled) &&
        object.status.numberReady == object.status.desiredNumberScheduled &&
        has(object.status.updatedNumberScheduled) &&
        object.status.updatedNumberScheduled == object.status.desiredNumberScheduled
      messageExpression: >-
        (has(object.status.numberReady) ? string(object.status.numberReady) : '0') +
        '/' + string(object.status.desiredNumberScheduled) + ' ready, ' +
        (has(object.status.updatedNumberScheduled) ? string(object.status.updatedNumberScheduled) : '0') + ' updated'
    - apiVersion: v1
      resources: [pods]
      ruleExpression: >-
        has(object.status) && has(object.status.phase) &&
        (object.status.phase == 'Running' || object.status.phase == 'Succeeded')
      messageExpression: "has(object.status.phase) ? 'phase: ' + object.status.phase : 'no phase'"
    - apiVersion: batch/v1
      resources: [jobs]
      ruleExpression: >-
        has(object.status) && has(object.status.conditions) &&
        object.status.conditions.exists(c, c.type == 'Complete' && c.status == 'True')
      messageExpression: "'job not complete'"
    - apiVersion: v1
      resources: [persistentvolumeclaims]
      ruleExpression: "has(object.status) && has(object.status.phase) && object.status.phase == 'Bound'"
      messageExpression: "has(object.status.phase) ? 'phase: ' + object.status.phase : 'no phase'"
    - apiVersion: v1
      resources: [persistentvolumes]
      ruleExpression: >-
        has(object.status) && has(object.status.phase) &&
        (object.status.phase == 'Bound' || object.status.phase == 'Available')
      messageExpression: "has(object.status.phase) ? 'phase: ' + object.status.phase : 'no phase'"

Save the example as convoy-defaults.yaml.
Apply it:
sh
```
kubectl apply -f convoy-defaults.yaml
```

2. Write a rule set for a custom resource#

A HealthRuleSet has a namespace. Only Releases in the same namespace can select it. This example gives a rule for the cert-manager Certificate. The rule passes when the Ready condition of the Certificate is True.

guides/healthruleset-certificate.yaml
# A namespaced HealthRuleSet for a custom resource: the cert-manager
# Certificate. A Certificate is healthy when its Ready condition is True.
# A Release in the same namespace selects it with
# spec.healthAssessment.localRuleSetSelector.matchLabels.team=platform.
apiVersion: extensions.convoy.kumodo.io/v1alpha1
kind: HealthRuleSet
metadata:
  name: cert-manager
  namespace: default
  labels:
    team: platform
spec:
  rules:
    - apiVersion: cert-manager.io/v1
      resources: [certificates]
      ruleExpression: >-
        has(object.status) && has(object.status.conditions) &&
        object.status.conditions.exists(c, c.type == 'Ready' && c.status == 'True')
      messageExpression: >-
        has(object.status) && has(object.status.conditions) &&
        object.status.conditions.exists(c, c.type == 'Ready')
        ? object.status.conditions.filter(c, c.type == 'Ready')[0].message
        : 'no Ready condition'

Set apiVersion to the group and version of your custom resource.
Set resources to the plural, lowercase resource name.
Write ruleExpression as a CEL expression that returns a bool.
Use has() guards for fields that can be missing.
Optionally, write messageExpression as a CEL expression that returns a string.
Add a label that the Release can select.

Apply the file:

kubectl apply -f healthruleset-certificate.yaml

The only CEL variable is object, the live resource. A rule applies to all resources with a matching apiVersion and resource name. The resource name of the object does not matter.

3. Select the rule sets in the Release#

guides/release-health-assessment.yaml
# A Release with health assessment. It selects the default GlobalHealthRuleSet
# (examples/health/convoy-defaults.yaml) and the HealthRuleSets with the label
# team=platform in its own namespace. A resource that fails a rule after the
# deadline is Degraded.
apiVersion: core.convoy.kumodo.io/v1alpha1
kind: Release
metadata:
  name: podinfo-health
  namespace: default
spec:
  clusterRef:
    name: dev-local
  source:
    type: gitRepository
    gitRepository:
      name: podinfo
      namespace: default
      ref:
        branch: master
  content:
    type: manifests
    manifests:
      paths:
        - kustomize/deployment.yaml
        - kustomize/service.yaml
  healthAssessment:
    globalRuleSetSelector:
      matchLabels:
        convoy.kumodo.io/default: "true"
    localRuleSetSelector:
      matchLabels:
        team: platform
    deadline: 2m
  syncPolicy:
    automatic: true
    prune:
      orphaned: false
      includeCRDs: false
      includeNamespaces: false
    interval: 1m

Set globalRuleSetSelector to select the default rule set.
Set localRuleSetSelector to select your HealthRuleSet.
Set deadline to the time that a resource can fail its rules before it is Degraded.

Apply the file:

kubectl apply -f release-health-assessment.yaml

A selector that is not set selects no rule set. An empty selector {} selects all rule sets. convoy uses the rules of all selected sets together.

The podinfo Release renders a Deployment and a Service. The default rules assess the Deployment. No rule applies to the Service, so the Service is Healthy. The Certificate rule applies only when a Release renders a Certificate.

Check the result#

Get the Release:
sh
```
kubectl get releases
```
Read the HEALTHY column of podinfo-health.

Show the state of each resource:

kubectl convoy release health podinfo-health

The command prints one line for each resource: the kind, the namespace and name, the state, and the rule message. It prints not assessed when the observer has no state yet.

Read the Healthy condition and its message:
sh
```
kubectl get release podinfo-health -o yaml
```
Make sure that Healthy changes to True with reason Healthy.

Before the first apply, the reason is Pending. While the Deployment rolls out, the reason is Progressing. A reason of Unknown with a message that starts with compiling health rules: means that a CEL expression does not compile. Fix the rule that the message names.

Healthy · written by convoy-observer (field manager convoy-health)

Reason	Status	Meaning
`Healthy`	True	All assessed resources are healthy.
`Progressing`	False	At least one resource is not healthy yet, and no resource is `Degraded` or `Unknown`.
`Degraded`	False	At least one resource is `Degraded`. It did not pass its health rules before `spec.healthAssessment.deadline`.
`Unknown`	Unknown	At least one resource has no health state, or the observer could not assess the Release (for example, a health rule does not compile).
`NotConfigured`	True	The Release has no `spec.healthAssessment`.
`Pending`	Unknown	The current render is not applied or not assessed yet. The health of the previous render does not count.

Clean up#

Delete the Release:
sh
```
kubectl delete release podinfo-health
```

Delete the HealthRuleSet:

kubectl delete healthruleset cert-manager

Keep the default GlobalHealthRuleSet if other Releases use it. Otherwise, delete it:
sh
```
kubectl delete globalhealthruleset convoy-defaults
```

Next steps#

Order resources with sync waves: use health as a gate between waves.
Roll back a Release: roll back when a rollout is not healthy.
Health rule sets: the fields of the rule sets.
Health assessment: how the observer assesses resources.

Add health rules to a Release

Before you start#

Steps#

1. Apply the default rule set#

2. Write a rule set for a custom resource#

3. Select the rule sets in the Release#

Check the result#

Clean up#

Next steps#

In the code