Lifecycle hooks

A lifecycle hook is a Job or a Pod that convoy creates on the target cluster at a fixed point of a sync or of a deletion. Use hooks for work that is not part of the rendered manifests. Examples are database migrations, smoke tests and notifications. You write the hooks in the top-level lifecycle field of the Release, or through the /lifecycle subresource. For the fields and admission rules, see Release lifecycle hooks.

Example#

apiVersion: core.convoy.kumodo.io/v1alpha1
kind: Release
metadata:
  name: podinfo-in-cluster-helm-with-lifecycle
  namespace: default
lifecycle:
  preSync:
    - name: pre-sleep
      type: Job
      timeout: 30s
      onFailure: Continue
      cleanup: Never
      deleteBeforeCreation: true
      jobTemplate:
        metadata:
          labels:
            test-label: "true"
          annotations:
            test-annotation: "true"
        spec:
          template:
            spec:
              restartPolicy: Never
              containers:
                - name: job-worker
                  image: "busybox:1.38.0"
                  command: ["sh", "-c", "sleep 10"]
  postSync:
    - name: post-sleep
      type: Job
      runAfter: Healthy
      timeout: 30s
      onFailure: Continue
      cleanup: OnSuccess
      jobTemplate:
        metadata:
          labels:
            test-label: "true"
          annotations:
            test-annotation: "true"
        spec:
          template:
            spec:
              restartPolicy: Never
              containers:
                - name: job-worker
                  image: "busybox:1.38.0"
                  command: ["sh", "-c", "sleep 10"]
mutations:
  - type: syncWave
    waveStrategy:
      strategy: waitForHealthy
      wave: 1
    target:
      group: ""
      version: v1
      kind: Service
spec:
  clusterRef:
    name: dev-local
  source:
    type: gitRepository
    gitRepository:
      name: podinfo
      namespace: default
      ref:
        branch: master
  content:
    type: helm
    helm:
      path: charts/podinfo
      releaseName: podinfo-helm
      values:
        - type: inline
          inline:
            probes:
              readiness:
                initialDelaySeconds: 15
              liveness:
                initialDelaySeconds: 15
  syncPolicy:
    automatic: true
    prune:
      orphaned: false
      includeCRDs: false
      includeNamespaces: false
    interval: 1m
  healthAssessment:
    globalRuleSetSelector:
      matchLabels:
        convoy.kumodo.io/default: "true"
    deadline: 1m

Phases#

convoy runs preSync, postSync and syncFailed hooks one time for each render. A new render runs them again. A reconcile of the same render does not run them again.

preSync#

While a preSync hook runs, convoy applies nothing. Applied is False with reason WaitingForHook. If the Release holds at a cluster wave, the preSync hooks do not start.

postSync and runAfter#

runAfter sets when a postSync hook starts. It is allowed only on postSync hooks.

• Applied (default): the hook starts immediately after the apply.
• Healthy: the hook starts when all resources of the Release are Healthy for the current render. This includes the resources of all waves.

Without spec.healthAssessment, Healthy has the same effect as Applied. runAfter only controls the start of the hook. convoy does not check it again after the hook starts.

A postSync hook does not change the result of the sync. The apply occurred before the hook.

syncFailed#

syncFailed hooks run when the render succeeded, but a step of the apply stage failed. Examples are a failed dry run, a failed apply, a failed prune, and a failed wave gate. They do not run when these fail:

• the fetch of the source, or the render
• the target cluster
• a preSync hook
• a cluster wave timeout (ClusterWaveFailed)

convoy starts syncFailed hooks and does not wait for them. Their result does not change the sync. onFailure has no effect.

preDelete and postDelete#

convoy runs the delete hooks also when spec.deletionPolicy.keepWorkloads is true. convoy skips the hooks in two cases:

• The Cluster object does not exist.
• The target cluster is not reachable, and blockOnClusterUnreachable is false.

With blockOnClusterUnreachable: true, an unreachable cluster blocks the deletion until it is reachable. See Deletion.

The workload#

Each hook has type: Job with a jobTemplate, or type: Pod with a podTemplate. A template has a partial metadata and a Kubernetes spec:

• metadata can set only namespace, labels and annotations. Without a namespace, convoy uses the target namespace of spec.content (releaseNamespace or targetNamespace). If that is also empty, convoy uses default.
• convoy sets the name of the workload. The name is {phase}-{name}-{hash}, for example presync-migrate-1f2e3d4c. The phase is in lowercase. convoy shortens the hook name to 32 characters. The hash comes from the Release name, the hook name and the render.
• convoy adds these labels. They replace your labels with the same key.

The workload is on a different cluster, so it has no owner reference. convoy finds the workloads of a hook with the release-uid, hook-phase and hook-name labels.

Templates and the if condition#

convoy renders the template as a Go template before it creates the workload. The if field is a CEL expression. Both use the same values:

lifecycle:
  postSync:
    - name: smoke-test
      type: Job
      if: 'cluster.tier == "production"'
      jobTemplate:
        spec:
          template:
            spec:
              restartPolicy: Never
              containers:
                - name: test
                  image: curlimages/curl:8.10.1
                  args: ["https://{{ .release.name }}.{{ .cluster.domain }}/healthz"]

• A key that does not exist renders as an empty value. It does not cause an error.
• An empty if is true.
• If if is false, or it cannot be evaluated, convoy skips the hook. It creates no workload and records no entry. A skipped hook is not in status.lifecycle.
• Admission does not compile the if expression or the template.

Mutations do not change hook templates. The mutation pipeline changes only the rendered objects.

Order in a phase#

weight puts the hooks of a phase in groups. convoy runs the groups in ascending order of weight:

1. convoy starts all hooks of the lowest group. Hooks with the same weight run in parallel.
2. convoy waits until all hooks of the group finish.
3. convoy cleans up the finished hooks and starts the next group.
4. When the last group finishes, the phase is complete.

Timeout#

timeout is the maximum time from the creation of the workload. When the timeout passes and the workload did not finish, the hook is TimedOut. Without timeout, convoy waits without a time limit.

convoy does not stop the workload at the timeout. The cleanup policy can delete it.

Failure#

A hook fails when the Job has the Failed condition, or the Pod has the Failed phase. A TimedOut hook is also a failure. onFailure sets what convoy does next:

convoy does not run a failed hook again for the same render. A new render runs it again.

Cleanup#

cleanup sets what convoy does with the workload when the hook finishes:

A hook that fails with onFailure: Fail stops the phase before the cleanup step. Thus, convoy does not delete its workload, also with cleanup: OnCompletion.

deleteBeforeCreation: true deletes all earlier workloads of the hook before convoy creates the new one. The name changes with each render. Thus, with cleanup: Never and deleteBeforeCreation: false, one workload for each render stays on the cluster.

There is no garbage collection of hook workloads. convoy does not delete the workloads that stay after the deletion of the Release. A retention policy is planned. Not built

Status#

convoy records each hook in status.lifecycle, in a list for each phase:

status:
  lifecycle:
    preSync:
      - name: migrate
        source: lifecycle
        kind: Job
        apiVersion: batch/v1
        resourceNamespace: default
        resourceName: presync-migrate-1f2e3d4c
        state: Failed
        renderRef: sha256:…
        followUpAction: Fail
        message: …

• state is Running, Succeeded, Failed or TimedOut.
• followUpAction is the onFailure value. convoy sets it only for a Failed or TimedOut hook. Without onFailure, the value is empty, and the hook acts as Continue.
• Each entry shows the last run of the hook. renderRef is the render of that run. An entry from an earlier render stays until the hook runs again.

The LifecycleSucceeded condition sums up all entries:

The condition reports a failure also when onFailure is Continue.

The Lifecycle printer column of kubectl get releases uses only the entries of the current render. Its value is N/A, Running:<phase>, Failed:<phase> or Completed:<phase>.

Helm chart hooks#

convoy does not translate Helm hooks into lifecycle hooks. It drops Helm test hooks. It applies other objects with a helm.sh/hook annotation as normal resources. A translation of Helm hooks into lifecycle hooks is planned. Not built