A failing mutation stops a pinned rollback and the rollout state machine

Behavior#

At each reconcile, convoy renders the source and runs the mutations. Then it resolves the pin and the rollout. A rollback pin and the last healthy render come from the render store, so they do not need the mutations. But when a mutation fails, the reconcile stops before the pin and before the rollout state machine.

Effect#

While a mutation fails, the Release reports Applied=False with reason DeployFailed or Unsupported, and convoy applies nothing:

• A pin to an earlier render does not apply.
• A Progressing rollout does not fail or roll back at its deadline.
• A RolledBack rollout does not apply the last healthy render again, so convoy does not undo drift on it.

The objects of the last apply stay on the cluster.

Workaround#

Change or remove the failing mutation with kubectl convoy release mutations set. Then the pin or the rollback applies at the next reconcile.

Fix direction#

Resolve a rollback pin and a RolledBack hold before the mutations, or skip the mutations when the render comes from the store.