Sharding

convoy divides its objects into a fixed number of shards. Each shard has a Kubernetes Lease. A replica of convoy-controller-manager or convoy-observer does the work of a shard only while it holds the lease of that shard. One replica holds all shards. Two or more replicas divide the shards, and a replica takes the shards of a replica that stops. Thus one mechanism gives high availability and scale. There is no separate leader election.

Shard key#

The shard of an object is FNV-1a("<namespace>/<name>") mod S. S is the shard count. The <namespace>/<name> is the shard key. The shard key depends on the kind:

All Releases that target one cluster are in the shard of that Cluster. Thus one replica handles a cluster, its Releases and their cluster waves. spec.clusterRef is immutable, so the shard of a Release does not change.

Shard label#

The apiserver writes the label convoy.kumodo.io/shard on each create and update of a Release, Cluster, GitRepository, HelmRegistry and OCIRegistry. The value is the shard index in decimal. The storage strategy of each kind writes the label in PrepareForCreate and PrepareForUpdate. It always replaces the value, so a user cannot change the shard of an object.

The controller-manager copies the label of the Release to its ReleaseInventory. The reconcile gate does not read the label. It computes the shard from the shard key. The label is necessary only for shard-scoped caches.

Shard count#

Set S with --shards on all three binaries. The default is 32.

• The apiserver uses --shards to compute the label.
• The controller-manager and the observer use --shards for the leases and for the reconcile gate.

All replicas of all three binaries must use the same value. A replica with a different value computes different shards. A change of S changes the shard of almost every object. The apiserver writes the new label only at the next write of each object. Thus a change of S is an operation on the full installation. More replicas than S do not add capacity, because the replicas after S own no shard.

Leases#

Each component uses two kinds of Lease objects in the lease namespace:

• <component> is convoy-cm for the controller-manager and convoy-obs for the observer.
• --replica-id sets the replica ID. The default is $POD_NAME, then the hostname.
• --lease-namespace sets the lease namespace. The default is $POD_NAMESPACE, then convoy-system.
• The labels sharding.convoy.kumodo.io/component and sharding.convoy.kumodo.io/role identify the leases of a component.

Assignment rule#

At each tick, the Coordinator of a replica does these steps:

1. It renews its member lease. If the lease does not exist, it creates it.
2. It lists the member leases of its component. A member is live when its lease renewal is not older than --lease-duration. The replica always counts itself as live.
3. It sorts the live replica IDs. Its own position in the list is k. The number of live replicas is N.
4. It wants each shard i where i mod N = k.
5. For each wanted shard, it creates, renews or takes the shard lease. It takes a lease only when the lease is free or expired.
6. For each shard that it owns but does not want, it frees the lease. It sets the holder to empty.
7. If the set of owned shards changed, it calls the OnChange callbacks.

The update of a shard lease uses the resourceVersion of the lease. If two replicas update one lease at the same time, one update fails with a conflict. The losing replica tries again at the next tick. A replica also deletes the member lease of another replica when that lease is older than 10 times --lease-duration.

If cm-c stops, N becomes 2. Then cm-a owns 0, 2, 4, 6 and cm-b owns 1, 3, 5, 7. The shard of each cluster does not change. Only the owner of the shard changes.

Coordinator lifecycle#

Both managers start the Coordinator in the same order:

1. They register an OnChange callback. The callback rebuilds the shard-scoped caches and adds the keys of all owned objects to the work queues.
2. Prime runs one tick synchronously, before the informers start. Thus the enqueue filters know the owned shards at the first informer event.
3. Run runs one tick, then one tick at each --lease-renew-interval.
4. When the context stops, Run frees all owned shard leases, with a time limit of 5 seconds. It does not delete the member lease.

Gates#

A replica checks ownership at two points:

• Enqueue filter. The informer event handler adds a key to the queue only when the replica owns the shard of the object. This filter only makes the queue shorter.
• Reconcile gate. At the start of each reconcile, the controller computes the shard again. If the replica does not own the shard, the reconcile returns without work and without a requeue. This gate gives the correctness, because the ownership can change while a key is in the queue.

The Cluster and source controllers use base.WithShardGate, which installs both checks on the object key. The Release controller computes the shard from spec.clusterRef. It installs its own enqueue filter and checks ownership in syncHandler, before the finalization. Thus only the owner finalizes a Release. The observer drift loop and health loop use the same rule.

The ownership check reads an in-memory set. It does not read the lease at each reconcile.

Components shard independently#

The controller-manager and the observer use the same shard key. They do not use the same leases. They can run different replica counts. A shard can have its controller-manager owner on one pod and its observer owner on a different pod. The two components communicate only through the API.

The source informers and the health rule set informers always cache all objects. A Release can reference a source or a rule set in any shard. A source controller verifies a source and writes its status only on the owner of the source shard. The fetch of the source content for a Release runs on the owner of the Release.

Failover timing#

The defaults are --lease-duration=30s and --lease-renew-interval=10s.

During a change of owner, a shard can have no owner for some ticks. The new owner reconciles all objects of the shard after it takes the lease, so no work is lost.

Set --lease-duration larger than the longest reconcile. Set --lease-renew-interval much smaller than --lease-duration. See Scaling and high availability.