Render storage

The render store keeps the content of each render that convoy applies. It is an object store outside etcd. etcd keeps only pointers to renders, for example status.lastSuccessful.renderRef. The object store keeps the render content and the ordered history. This page describes the implementation. For the user-level behavior, see Pin and history. For the backend URLs and credentials, see Render store backends.

Layers#

The store has two layers:

blobstore.Open opens a gocloud bucket from the --render-store-url value. The registered gocloud drivers are s3, gs, azblob, file and mem. List returns all keys under a prefix in lexical order, without a delimiter. Delete of an absent key is not an error. Get of an absent key returns blobstore.ErrNotFound.

The RenderStore does not know about Kubernetes types. A history entry has only three fields: renderRef, syncedAt and summary.

Key layout#

All keys of a Release are under the prefix <namespace>/<name>/. There is no global prefix. To isolate two convoy installations, use two buckets.

• renderRef is sha256: and the hex SHA-256 of the JSON of each rendered object, in render order.
• <time> is the UTC sync time in the layout 20060102T150405.000000000Z. The layout has a fixed width, so the lexical order of the keys is the chronological order.
• The time contains no -, and the renderRef contains no -. The store finds the renderRef of an entry from its key, before the first -. It does not read the entry for this.

Write-once blobs#

The key of a blob is the hash of its content. PutRender first calls Exists. If the blob exists, PutRender does not write it again. Thus convoy never changes a stored blob. It only reads or deletes it.

Persist before apply#

The controller-manager writes the blob before it applies an object. The order in a sync is:

1. The pre-apply dry run. A render that fails the dry run is not stored.
2. persistRender writes the blob. If this write fails, convoy applies nothing. The reason is DeployFailed, and the controller retries.
3. The preSync hooks, the sync waves, the prune and the postSync hooks.
4. recordHistory adds a history entry.
5. runGC runs the garbage collection.

Steps 4 and 5 run only after a full success of step 3. A sync that stops at a wave gate, a hook or an error has a stored blob but no history entry. The garbage collection keeps that blob, because status.lastAttempted refers to it. See the protected renders below.

History entries#

recordHistory first calls HasHistoryRef. This call lists the history keys and compares the renderRef of each key. If an entry for the render exists, recordHistory does not add one. The history is a set of different renders, in the order of their first apply.

recordHistory does not fail the sync. If the store returns an error, the controller logs it. The apply stays successful, and the history has a gap.

A rollback by spec.failurePolicy does not call recordHistory and does not call runGC. See Rollout state machine.

Garbage collection#

runGC reads the effective keepHistory. An unset value gives 10. A value of 0 or less stops the garbage collection, so convoy keeps all entries and all blobs.

For a positive value N, RenderStore.GC does these steps:

1. It lists the history keys and deletes the oldest keys until N keys remain.
2. It lists the history keys again and collects their renderRef values.
3. It adds the protected renders to this set.
4. It deletes each blob whose renderRef is not in the set.

convoy reads the protected renders from the Release at the start of the reconcile:

• the render that this sync applied
• status.lastSuccessful.renderRef
• status.lastAttempted.renderRef
• pin.renderRef
• status.lastHealthy.renderRef

The protection of status.lastAttempted covers the time between step 2 and step 4 of the sync. The protection of status.lastHealthy keeps the rollback target. If the garbage collection returns an error, the controller logs it and does not fail the reconcile.

Deletion of a Release#

When convoy finalizes a Release, it calls DeleteRelease after it deletes the ReleaseInventory. DeleteRelease lists all keys under <namespace>/<name>/ and deletes each key. It skips a key that does not start with the prefix. If a delete fails, convoy keeps the finalizer and retries. See Deletion.

Observer reads#

The observer opens its own store from its own --render-store-url. It only reads:

If pinnedRender or rolledBackRender does not find the blob, the observer renders the source instead. A mem:// store is private to one process, so the observer cannot read the blobs of the controller-manager.

Without a render store#

An empty --render-store-url gives a nil RenderStore. Each caller checks for nil:

• persistRender, recordHistory and runGC do nothing.
• status.resolvedRetentionPolicy.historyCount is 0.
• A pin to a render other than the current render fails with reason Pinned.
• applyLastHealthy returns without an apply.
• /history returns 503. /diff?render= returns an error.

Single writer#

The sharding gives each Release to one controller-manager replica. Only that replica writes the keys of the Release, so the history appends do not race. The store has no lock of its own. See Sharding.