Run a database migration and a smoke test with lifecycle hooks

Run a migration Job before convoy applies a render, and a smoke test after the Release is healthy.

This guide adds two lifecycle hooks to a Release. A preSync hook runs a database migration before convoy applies the render. If the migration fails, convoy applies nothing. A postSync hook runs a smoke test after all resources of the Release are healthy. You write the hooks through the /lifecycle subresource with the kubectl convoy plugin. For the full behavior, see Lifecycle hooks.

Before you start#

Make sure that the GitRepository podinfo and the Cluster dev-local exist. See Deploy from Git.
Make sure that a GlobalHealthRuleSet with the label convoy.kumodo.io/default: "true" exists. See Add health rules to a Release.
Save each example under the file name in its title bar. The commands use these file names.

Steps#

1. Deploy the Release with example hooks#

The example has a preSync hook and a postSync hook. Both hooks run sleep 10. The Release has spec.healthAssessment, so runAfter: Healthy waits for healthy resources.

releases/release-gitrepository-helm-in-cluster-with-lifecycle.yaml
apiVersion: core.convoy.kumodo.io/v1alpha1
kind: Release
metadata:
  name: podinfo-in-cluster-helm-with-lifecycle
  namespace: default
lifecycle:
  preSync:
    - name: pre-sleep
      type: Job
      timeout: 30s
      onFailure: Continue
      cleanup: Never
      deleteBeforeCreation: true
      jobTemplate:
        metadata:
          labels:
            test-label: "true"
          annotations:
            test-annotation: "true"
        spec:
          template:
            spec:
              restartPolicy: Never
              containers:
                - name: job-worker
                  image: "busybox:1.38.0"
                  command: ["sh", "-c", "sleep 10"]
  postSync:
    - name: post-sleep
      type: Job
      runAfter: Healthy
      timeout: 30s
      onFailure: Continue
      cleanup: OnSuccess
      jobTemplate:
        metadata:
          labels:
            test-label: "true"
          annotations:
            test-annotation: "true"
        spec:
          template:
            spec:
              restartPolicy: Never
              containers:
                - name: job-worker
                  image: "busybox:1.38.0"
                  command: ["sh", "-c", "sleep 10"]
mutations:
  - type: syncWave
    waveStrategy:
      strategy: waitForHealthy
      wave: 1
    target:
      group: ""
      version: v1
      kind: Service
spec:
  clusterRef:
    name: dev-local
  source:
    type: gitRepository
    gitRepository:
      name: podinfo
      namespace: default
      ref:
        branch: master
  content:
    type: helm
    helm:
      path: charts/podinfo
      releaseName: podinfo-helm
      values:
        - type: inline
          inline:
            probes:
              readiness:
                initialDelaySeconds: 15
              liveness:
                initialDelaySeconds: 15
  syncPolicy:
    automatic: true
    prune:
      orphaned: false
      includeCRDs: false
      includeNamespaces: false
    interval: 1m
  healthAssessment:
    globalRuleSetSelector:
      matchLabels:
        convoy.kumodo.io/default: "true"
    deadline: 1m

Apply the Release.

kubectl apply -f releases/release-gitrepository-helm-in-cluster-with-lifecycle.yaml

Monitor the Release.

kubectl get releases podinfo-in-cluster-helm-with-lifecycle -w

Examine the LIFECYCLE column. The value changes from Running:PreSync to Running:PostSync, and then to Completed:PostSync.

2. Write the hooks file#

The file contains a ReleaseLifecycle object. It is not a Kubernetes object. The top-level keys are the phases. Save this content as hooks.yaml:

yaml
preSync:
  - name: migrate
    type: Job
    timeout: 5m
    onFailure: Fail
    cleanup: OnSuccess
    deleteBeforeCreation: true
    jobTemplate:
      spec:
        backoffLimit: 1
        template:
          spec:
            restartPolicy: Never
            containers:
              - name: migrate
                image: busybox:1.38.0
                command: ["sh", "-c", "echo running database migrations"]
postSync:
  - name: smoke-test
    type: Job
    runAfter: Healthy
    timeout: 2m
    onFailure: Continue
    cleanup: OnSuccess
    jobTemplate:
      spec:
        backoffLimit: 0
        template:
          spec:
            restartPolicy: Never
            containers:
              - name: smoke-test
                image: curlimages/curl:8.10.1
                args: ["-fsS", "http://podinfo-helm:9898/healthz"]

Replace the image and the command of migrate with your migration tool.
Keep onFailure: Fail on migrate. A failed migration then stops the sync before convoy applies an object.
Keep runAfter: Healthy on smoke-test. The test then starts after all resources are healthy for the current render.
Set a low backoffLimit. A Job fails only after all its retries fail.

The hook workload runs in the target namespace of spec.content. Without a target namespace, it runs in default. Thus the smoke test can use the short Service name podinfo-helm.

3. Select the cleanup behavior#

cleanup and deleteBeforeCreation control the hook workloads on the target cluster:

Setting	Effect	Use it when
`cleanup: Never` (default)	convoy does not delete the workload.	You want to keep all logs.
`cleanup: OnSuccess`	convoy deletes the workload after a success.	You want to keep the logs of a failure only.
`cleanup: OnCompletion`	convoy deletes the workload after a success or a failure.	You do not need the logs.
`deleteBeforeCreation: true`	convoy deletes the earlier workloads of the hook before it creates a new one.	You want one workload for each hook on the cluster.

A hook that fails with onFailure: Fail stops the phase before the cleanup step. Thus its workload stays, also with cleanup: OnCompletion.
convoy does not collect old hook workloads. With cleanup: Never and deleteBeforeCreation: false, one workload stays for each render.

4. Set the hooks#

Replace the hooks of the Release.

kubectl convoy release lifecycle set podinfo-in-cluster-helm-with-lifecycle -f hooks.yaml

Read the hooks back.

kubectl convoy release lifecycle get podinfo-in-cluster-helm-with-lifecycle

lifecycle set replaces all phases. convoy clears each phase that is not in the file. The new hooks have new names, so convoy runs them also for the current render.

5. Monitor the hooks#

Monitor the Release.

kubectl get releases podinfo-in-cluster-helm-with-lifecycle -w

Examine the LIFECYCLE column. The value is Running:PreSync while migrate runs.

List the hook workloads.

sh
kubectl get jobs -n default -l convoy.kumodo.io/release-name=podinfo-in-cluster-helm-with-lifecycle

The workload names have the format {phase}-{name}-{hash}, for example presync-migrate-1f2e3d4c.

Check the result#

Read the hook records.

sh
kubectl get releases podinfo-in-cluster-helm-with-lifecycle -o jsonpath='{.status.lifecycle}'

Find the entries migrate in preSync and smoke-test in postSync. Each entry must have state: Succeeded.
Find the LifecycleSucceeded condition. Its status must be True with reason AllSucceeded.
sh
```
kubectl describe releases podinfo-in-cluster-helm-with-lifecycle
```
Examine the LIFECYCLE column. The value must be Completed:PostSync.

A hook with an if expression that is false does not run. It has no entry in status.lifecycle.

If migrate fails, you see these values:

LIFECYCLE is Failed:PreSync.
Applied is False with reason HookFailed.
status.lastAttempted.failure.stage is HookFailed, and hookName is migrate.
The Job of migrate stays on the cluster. Read its logs to find the cause.

convoy does not run a failed hook again for the same render. Correct the cause, then push a new revision to the source. The new render runs the hooks again.

LifecycleSucceeded · written by convoy-controller-manager (field manager convoy-release)

Reason	Status	Meaning
`AllSucceeded`	True	All lifecycle hooks succeeded or were skipped.
`Running`	Unknown	At least one lifecycle hook is still running.
`HookFailed`	False	A lifecycle hook failed. This is so for each `onFailure` value. `onFailure` only controls if the sync continues.

Clean up#

To run hooks when you delete a Release, add preDelete and postDelete hooks. This example shows the two phases:

releases/release-gitrepository-helm-in-cluster-with-deletion-lifecycle.yaml
apiVersion: core.convoy.kumodo.io/v1alpha1
kind: Release
metadata:
  name: podinfo-in-cluster-helm-with-deletion-lifecycle
  namespace: default
lifecycle:
  preDelete:
    - name: pre-sleep
      type: Job
      timeout: 30s
      onFailure: Continue
      cleanup: OnSuccess
      jobTemplate:
        metadata:
          labels:
            test-label: "true"
          annotations:
            test-annotation: "true"
        spec:
          template:
            spec:
              restartPolicy: Never
              containers:
                - name: job-worker
                  image: "busybox:1.38.0"
                  command: ["sh", "-c", "sleep 10"]
  postDelete:
    - name: post-sleep
      type: Job
      timeout: 30s
      onFailure: Continue
      cleanup: OnSuccess
      jobTemplate:
        metadata:
          labels:
            test-label: "true"
          annotations:
            test-annotation: "true"
        spec:
          template:
            spec:
              restartPolicy: Never
              containers:
                - name: job-worker
                  image: "busybox:1.38.0"
                  command: ["sh", "-c", "sleep 10"]
mutations:
  - type: syncWave
    waveStrategy:
      strategy: waitForHealthy
      wave: 1
    target:
      group: ""
      version: v1
      kind: Service
spec:
  clusterRef:
    name: dev-local
  source:
    type: gitRepository
    gitRepository:
      name: podinfo
      namespace: default
      ref:
        branch: master
  content:
    type: helm
    helm:
      path: charts/podinfo
      releaseName: podinfo-helm
      values:
        - type: inline
          inline:
            probes:
              readiness:
                initialDelaySeconds: 15
              liveness:
                initialDelaySeconds: 15
  syncPolicy:
    automatic: true
    prune:
      orphaned: false
      includeCRDs: false
      includeNamespaces: false
    interval: 1m
  healthAssessment:
    globalRuleSetSelector:
      matchLabels:
        convoy.kumodo.io/default: "true"
    deadline: 1m

Delete the Release.

kubectl delete releases podinfo-in-cluster-helm-with-lifecycle

List the hook workloads that stay on the cluster.

sh
kubectl get jobs -n default -l convoy.kumodo.io/release-name=podinfo-in-cluster-helm-with-lifecycle

Delete these workloads.

sh
kubectl delete jobs -n default -l convoy.kumodo.io/release-name=podinfo-in-cluster-helm-with-lifecycle

Next steps#

Read Lifecycle hooks for the if condition, weights and timeouts.
Read Release lifecycle hooks for all fields.
Order the resources of a Release with sync waves.
Return to a known render with Roll back a Release.

Run a database migration and a smoke test with lifecycle hooks

Before you start#

Steps#

1. Deploy the Release with example hooks#

2. Write the hooks file#

3. Select the cleanup behavior#

4. Set the hooks#

5. Monitor the hooks#

Check the result#

Clean up#

Next steps#

In the code