KB-7 Suspected, not reproduced
A wave timeout can count from the apply time of an earlier render
Behavior#
A bounded wave gate (waitForHealthyWithTimeout) counts its time limit from
status.waves[].startedAt of the previous wave. convoy keeps startedAt by wave number across
reconciles, and does not reset it for a new render.
Effect#
- After a new render, a bounded gate can count from the first apply of an earlier render. The gate can time out at once.
- A Release whose rollout already succeeded can fail or roll back later. When an earlier wave becomes unhealthy, its
startedAtis old, so the time sincestartedAtis already more than the time limit.
Workaround#
Use waitForHealthy without a time limit, or rely on spec.healthAssessment.deadline and
failurePolicy.rollback: OnReleaseFailure.
Fix direction#
Reset the wave timestamps when the render changes.
In the code
convoy v0.9.0
internal/controller/release.(*Controller).stageSyncinternal/controller/release/controller.go:1092internal/controller/release.(*Controller).waveHealthGateinternal/controller/release/controller.go:1593