How to Use Kubernetes LeaderElection for Custom Controller High Availability

In Kubernetes, high availability and fault tolerance are essential for system reliability. For controllers, LeaderElection is a mechanism that ensures only one instance of a controller operates on a specific task at a time in a multi-replica deployment. This blog delves into the concept of LeaderElection, its importance, implementation, and best practices.

---

What is Kubernetes LeaderElection?

LeaderElection is a process where multiple replicas of a controller or service coordinate to elect a single leader that performs the primary tasks, while others remain on standby. If the leader fails, another instance is elected to ensure continuity.

Why is LeaderElection Necessary?

• Prevents duplicate work: Without a leader, multiple controller replicas could simultaneously act on the same resource, leading to conflicts or inconsistencies.
• Ensures high availability: If the leader fails, a new one is promptly elected, maintaining uninterrupted operation.

Kubernetes LeaderElection Best Practices: Achieving Reliable Controller Management

---

How LeaderElection Works

LeaderElection relies on coordination primitives provided by Kubernetes, typically using ConfigMaps or Leases stored in the API server.

1. Lease-based LeaderElection:
   The leader acquires a lease by updating a resource (like a ConfigMap or Lease object) with its identity and timestamp.
2. Health checks:
   The leader continuously updates its lease to indicate it is active.
3. Failover:
   If the leader fails to update the lease within the specified timeout, other candidates compete to acquire the lease.

---

Key Components of LeaderElection

1. LeaderElectionConfiguration

A configuration block for enabling leader election in custom controllers or operators.

Example configuration:

leaderElection: true
leaderElectionID: "my-custom-controller-leader-election"
leaderElectionNamespace: "kube-system"
leaseDuration: 15s
renewDeadline: 10s
retryPeriod: 2s

2. Leases API

The Lease resource in the coordination.k8s.io API group is often used for LeaderElection.

Example Lease Object:

apiVersion: coordination.k8s.io/v1
kind: Lease
metadata:
  name: my-leader-election-lease
  namespace: kube-system
spec:
  holderIdentity: instance-1
  leaseDurationSeconds: 15
  renewTime: 2024-01-01T00:00:00Z

---

How to Implement LeaderElection in Go

LeaderElection can be added to custom controllers using the Kubernetes client-go library.

Setup Code for LeaderElection

1. Import Required Libraries:

import (
    "context"
    "fmt"
    "os"
    "time"

    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/rest"
    "k8s.io/client-go/tools/leaderelection"
    "k8s.io/client-go/tools/leaderelection/resourcelock"
)

2. Create a Resource Lock:
   The resourcelock package provides abstractions for Lease or ConfigMap-based locks.

config, err := rest.InClusterConfig()
if err != nil {
    panic(err)
}

clientset, err := kubernetes.NewForConfig(config)
if err != nil {
    panic(err)
}

lock, err := resourcelock.New(
    resourcelock.LeasesResourceLock,
    "kube-system", // Namespace
    "my-controller", // Lease name
    clientset.CoreV1(),
    clientset.CoordinationV1(),
    resourcelock.ResourceLockConfig{
        Identity: os.Hostname(),
    },
)
if err != nil {
    panic(err)
}

3. Start LeaderElection:

leaderelection.RunOrDie(context.TODO(), leaderelection.LeaderElectionConfig{
    Lock:          lock,
    LeaseDuration: 15 * time.Second,
    RenewDeadline: 10 * time.Second,
    RetryPeriod:   2 * time.Second,
    Callbacks: leaderelection.LeaderCallbacks{
        OnStartedLeading: func(ctx context.Context) {
            fmt.Println("Started leading")
            // Your controller's main logic
        },
        OnStoppedLeading: func() {
            fmt.Println("Stopped leading")
        },
        OnNewLeader: func(identity string) {
            if identity == os.Hostname() {
                fmt.Println("I am the leader now")
            } else {
                fmt.Printf("New leader elected: %s\n", identity)
            }
        },
    },
})

---

Testing LeaderElection

1. Deploy your controller with multiple replicas:

   spec:
     replicas: 3

2. Verify logs to see which instance becomes the leader.
3. Simulate leader failure by terminating the leader pod and observe failover.

---

Best Practices for LeaderElection

1. Use short timeouts carefully:
   Setting a very short lease duration or renew deadline may lead to unnecessary failovers due to temporary network issues.

2. Avoid leader-specific data persistence:
   If the leader persists state, ensure it is accessible to other instances after a failover.

3. Monitor LeaderElection health:
   Use metrics and logs to monitor the status of LeaderElection in your cluster.

4. Leverage Kubernetes RBAC:
   Secure the resources (e.g., Lease or ConfigMap) used for LeaderElection to prevent unauthorized access.

---

Example Use Cases for LeaderElection

1. Custom Operators:
   Ensures only one operator instance performs resource reconciliation.

2. Backup Jobs:
   Ensures only one instance performs a backup at a time.

3. Distributed Systems Coordination:
   Facilitates leader selection in distributed systems for tasks like coordination or consensus.

---

Conclusion

LeaderElection is a vital mechanism in Kubernetes for ensuring high availability and preventing conflicts in multi-replica deployments. By following this guide, you can implement LeaderElection in your custom controllers, enhancing their reliability and fault tolerance.

What use cases do you have in mind for LeaderElection? Share your thoughts in the comments!