Roll back a Release

This guide shows two ways to return a Release to an earlier render. In part 1, you select a render from the history and pin the Release to it. In part 2, convoy detects a rollout that does not become healthy before a deadline, and applies the last healthy render. The two ways read the renders from the render store. For the full behavior, see Pin and history and Progressive rollout.

Before you start#

• Configure a render store. Set the same --render-store-url on convoy-controller-manager and on convoy-observer. See Render store backends.
• Do not use mem://. The observer cannot read the renders of the controller-manager from it.
• Make sure that the GitRepository podinfo and the Cluster dev-local exist. See Deploy from Git.
• Make sure that a GlobalHealthRuleSet with the label convoy.kumodo.io/default: "true" exists. See Add health rules to a Release.
• Save each example under the file name in its title bar. The commands use these file names.

Without a render store, history returns 503 Service Unavailable. A pin to an earlier render fails, and an automatic rollback cannot occur.

Steps#

Part 1: Roll back manually with a pin#

1. Deploy the first render#

The Release has spec.healthAssessment and spec.failurePolicy. Part 2 uses these fields.

apiVersion: core.convoy.kumodo.io/v1alpha1
kind: Release
metadata:
  name: podinfo-rollback
  namespace: default
spec:
  clusterRef:
    name: dev-local
  source:
    type: gitRepository
    gitRepository:
      name: podinfo
      namespace: default
      ref:
        branch: master
  content:
    type: helm
    helm:
      path: charts/podinfo
      releaseName: podinfo-rollback
      values:
        - type: inline
          inline:
            image:
              tag: 6.14.0
            replicaCount: 1
  syncPolicy:
    automatic: true
    interval: 1m
  healthAssessment:
    globalRuleSetSelector:
      matchLabels:
        convoy.kumodo.io/default: "true"
    deadline: 2m
  failurePolicy:
    rollback: OnReleaseFailure

1. Apply the Release.

   sh

   kubectl apply -f guides/release-podinfo-rollback.yaml

2. Monitor the Release until STATE is Succeeded.

   sh

   kubectl get releases podinfo-rollback -w

2. Deploy a second render#

This version sets replicaCount: 2. The render changes, so it gets a new renderRef.

apiVersion: core.convoy.kumodo.io/v1alpha1
kind: Release
metadata:
  name: podinfo-rollback
  namespace: default
spec:
  clusterRef:
    name: dev-local
  source:
    type: gitRepository
    gitRepository:
      name: podinfo
      namespace: default
      ref:
        branch: master
  content:
    type: helm
    helm:
      path: charts/podinfo
      releaseName: podinfo-rollback
      values:
        - type: inline
          inline:
            image:
              tag: 6.14.0
            replicaCount: 2
  syncPolicy:
    automatic: true
    interval: 1m
  healthAssessment:
    globalRuleSetSelector:
      matchLabels:
        convoy.kumodo.io/default: "true"
    deadline: 2m
  failurePolicy:
    rollback: OnReleaseFailure

1. Apply the second version.

   sh

   kubectl apply -f guides/release-podinfo-rollback-v2.yaml

2. Monitor the Release until STATE is Succeeded again.

3. Find the render in the history#

1. Show the history.

   sh

   kubectl convoy release history podinfo-rollback

2. Find the two entries. Each line has the time of the first apply, the renderRef and the source summary. The newest entry is first.

3. Copy the renderRef of the second line. This is the first render.

4. Examine the change#

1. Compare the first render with the live objects.

   sh

   kubectl convoy release diff podinfo-rollback --render sha256:…

2. Find the Deployment. The output shows spec.replicas with the live value 2 and the stored value 1.

The output shows what the rollback changes. For the full result, add -o json.

5. Pin the first render#

1. Pin the Release to the first render.

   sh

   kubectl convoy release pin podinfo-rollback --render-ref sha256:…

2. Read the pin status.

   sh

   kubectl get releases podinfo-rollback -o jsonpath='{.status.pin.intent}'

3. Examine the value. It is rollback until convoy applies the pinned render. Then it is freeze.

The pin starts a reconcile immediately. convoy reads the pinned render from the store and applies it with the usual apply: dry run, hooks, sync waves and prune. While the pin is set, convoy does not run the rollout state machine. A change to the source does not change the cluster.

If the pinned render is not in the store, Applied is False with reason Pinned. convoy applies nothing.

6. Clear the pin#

1. Clear the pin.

   sh

   kubectl convoy release unpin podinfo-rollback

2. Make sure that convoy applies the current render again. spec.replicas of the Deployment changes to 2.

Part 2: Let convoy roll back a failed rollout#

7. Examine the failure policy#

The Release from part 1 has these fields:

spec:
  healthAssessment:
    deadline: 2m
  failurePolicy:
    rollback: OnReleaseFailure

• deadline is the maximum time for a new render to become healthy. The time starts at status.rollout.startedAt.
• rollback: OnReleaseFailure applies the last healthy render when the deadline passes.
• A rollback value other than Never requires spec.healthAssessment.deadline. The API rejects a Release without it.

8. Deploy a render that fails#

This version uses an image tag that does not exist. The new pods cannot start.

apiVersion: core.convoy.kumodo.io/v1alpha1
kind: Release
metadata:
  name: podinfo-rollback
  namespace: default
spec:
  clusterRef:
    name: dev-local
  source:
    type: gitRepository
    gitRepository:
      name: podinfo
      namespace: default
      ref:
        branch: master
  content:
    type: helm
    helm:
      path: charts/podinfo
      releaseName: podinfo-rollback
      values:
        - type: inline
          inline:
            image:
              tag: 0.0.0-does-not-exist
            replicaCount: 2
  syncPolicy:
    automatic: true
    interval: 1m
  healthAssessment:
    globalRuleSetSelector:
      matchLabels:
        convoy.kumodo.io/default: "true"
    deadline: 2m
  failurePolicy:
    rollback: OnReleaseFailure

1. Apply the broken version.

   sh

   kubectl apply -f guides/release-podinfo-rollback-broken.yaml

2. Monitor the Release.

   sh

   kubectl get releases podinfo-rollback -w

3. Examine the STATE column. It is Progressing until the deadline passes. Then it is RolledBack.

The rollback applies the last healthy render. This apply is smaller than a usual apply:

• It applies all objects in one step. It does not use sync waves.
• It does not run the dry run or the lifecycle hooks.
• It does not add a history entry.

9. Start the rollout again#

A RolledBack rollout stays until a new rollout starts. Do the checks of “Check the result” before this step. Then select one of two options:

• The cause is in the manifest: correct the manifest and apply it. A new render starts a new rollout.
• The cause is outside the render, for example a missing image: correct the cause, then send a forced sync.

To send a forced sync, do these steps:

1. Send a forced sync.

   sh

   kubectl convoy release sync podinfo-rollback --force

2. Make sure that STATE changes to Progressing.

A forced sync skips the pre-apply dry run. It starts a new rollout from Failed or RolledBack. If the cause is still there, the rollout fails again at the deadline.

Check the result#

1. Read the rollout status.

   sh

   kubectl get releases podinfo-rollback -o jsonpath='{.status.rollout}'

2. After the rollback, make sure that state is RolledBack. message names the render that convoy applied.

3. Compare this render with status.lastHealthy.renderRef. The two values must be the same.

4. Make sure that the RolloutSucceeded condition is False.

   sh

   kubectl describe releases podinfo-rollback

5. Make sure that Applied is True with reason RolledBack.

6. Make sure that the pods of the Deployment use the image tag 6.14.0.

Clean up#

1. Delete the Release.

   sh

   kubectl delete releases podinfo-rollback

convoy deletes the keys of the Release in the render store before it clears the finalizer.

Next steps#

• Read Pin and history for the retention of renders.
• Read Progressive rollout for the rollout states.
• Roll back on a sync wave timeout with OnWaveFailure. See Sync waves.
• Write the health rules that decide the rollout. See Health rules.