TiDB Failover with TiDB Operator

• Objective: Learn TiDB Failover with TiDB Operator
• Prerequisites:
  • background knowledge of TiDB
  • background knowledge of Kubernetes and TiDB Operator
  • AWS account
  • TiDB cluster on AWS
• Optionality: Required
• Estimated time: 30min

This document focuses on demonstrating the failover of TiDB with TiDB Operator. One design goal of TiDB Operator is to automatically handle failures.

Prepare

Note

We must have enough avaiable nodes for the cluster to have a spare; for example, if we have 3 TiKV pods, we must prepare at least 4 Kubernetes nodes.

If you've deployed using Terraform, you should adjust the default_cluster_tikv_count and default_cluster_pd_count variables accordingly.

To demonstrate failover, you should put some load on your cluster. For instructions on how to run sysbench, refer to Run Sysbench.

TiKV Failover

In this section, we will explore the failover scenarios for TiKV. TiKV uses the Raft consensus algorithm to achieve high availability. Data stored in TikV has multiple replicas.

By default, each data has 3 replicas and each replica is placed in 3 TiKV pods. As long as 2 replicas are available, there should be no impact to the sysbench workload.

On AWS, EC2 instances can be provisioned in different Availability Zones (AZ). TiDB Operator schedules TiKV pods in different Availability Zones for high availability. For example, if there are 3 TiKV pods, the 3 TiKV pods will be scheduled on nodes in different AZs.

List Pods

You can use the following command to list all pods for the TiDB cluster.

kubectl get pod -n ${namespace}

NAME                                    READY   STATUS    RESTARTS   AGE
my-cluster-discovery-85cbf5454b-x7gjl   1/1     Running   0          10h
my-cluster-pd-0                         1/1     Running   0          3d17h
my-cluster-pd-1                         1/1     Running   0          3d17h
my-cluster-pd-2                         1/1     Running   0          3d17h
my-cluster-tidb-0                       2/2     Running   0          3d18h
my-cluster-tidb-1                       2/2     Running   0          3d18h
my-cluster-tikv-0                       1/1     Running   0          3d12h
my-cluster-tikv-1                       1/1     Running   0          3d12h
my-cluster-tikv-2                       1/1     Running   0          3d12h

Kill One TiKV Pod

You can use the following command to delete one TiKV pod:

kubectl delete pod my-cluster-tikv-2 -n ${namespace}

pod "my-cluster-tikv-2" deleted

After TiKV pod deletion, we can check the status again and we can see that a new TiKV pod with the same name is created by TiDB Operator. Note that the age of my-cluster-tikv-2 should be a small number, usually a couple of seconds.

kubectl get pod my-cluster-tikv-2 -n ${namespace}

NAME                READY    STATUS    RESTARTS   AGE
my-cluster-tikv-2   1/1      Running   0          3s

In addition, the sysbench workload is not affected.

Kill Two TiKV Pods

You can use the following command to delete two TiKV pods:

kubectl delete pod my-cluster-tikv-1 my-cluster-tikv-2 -n ${namespace}

pod "my-cluster-tikv-1" deleted
pod "my-cluster-tikv-2" deleted

After TiKV pod deletion, we can check the status again and we can found that the pod deleted will be recreated by TiDB Operator. Note that the age of my-cluster-tikv-1 and my-cluster-tikv-2 should be a small number, usually a couple of seconds.

In this case, however, the QPS of sysbench workload will drop to 0 and return to normal after a few seconds.

Delete One TiKV Node

Configure TiKV

• Optionality: Optional

PD marks a TiKV store as down after 30 minutes by default. To reduce the wait time, you can configure max-store-down-time in TiDBCluster CR:

kubectl edit tc my-cluster -n ${namespace}

You should set pd.spec.config.schedule.max-store-down-time to 5m.

Node Deletion

You can delete TiKV nodes in AWS console:

After deleting one TiKV node, the sysbench's QPS will decrease and the status of one TiKV pod will change to pending as there are no available machines to schedule the pod on.

To allow TiDB operator to schedule the pending pod, we need to add new TiKV nodes. You can add new TiKV nodes in AWS console:

Note

As TiDB Operator schedules TiKV pods in different Availability Zones, ensure that the AZ of the new node matches the one that has been deleted.

After waiting for max-store-down-time5 + tikvFailoverPeriod (30m + 5m by default), TiDB Operator creates a new TiKV pod and data will be moved to the new pod.

Cleanup

If the failed TiKV pod comes back online, TiDB Operator does not automatically delete the newly created node, and you need to manually drop it and restore the original number of nodes.

To do this, you can delete the TiKV node from the status.tikv.failureStores field of the TidbCluster CR.

Make One TiKV Pod Continuously Unavailable

We can use kubectl patch command to update the image of a TiKV pod and point to a fake image:

kubectl patch pod my-cluster-tikv-2 -n ${namespace} -p '{"spec":{"containers":[{"name":"tikv","image":"hub.pingcap.net/pingcap/fake"}]}}'

After executing the above command, the status of the TiKV pod will change to ErrImagePull:

kubectl get pod my-cluster-tikv-2 -n ${namespace}

NAME                READY   STATUS         RESTARTS   AGE
my-cluster-tikv-2   0/1     ErrImagePull   0          57m

The above operation makes one TiKV pod stays in a failure state. After waiting for max-store-down-time5 + tikvFailoverPeriod (30m + 5m by default), TiDB Operator creates a new TiKV pod and data will be moved to the new pod.

Cleanup

If the failed TiKV pod comes back online, TiDB Operator does not automatically delete the newly created node, and you need to manually drop it and restore the original number of nodes.

To do this, you can delete the TiKV node from the status.tikv.failureStores field of the TidbCluster CR.

PD Failover

In this section, we demonstrate how TiDB operator handles PD failures.

Make one PD pod continuously unavailable

We can use kubectl patch command to update the image of PD pod:

kubectl patch pod my-cluster-pd-2 -n ${namespace} -p '{"spec":{"containers":[{"name":"pd","image":"hub.pingcap.net/pingcap/fake"}]}}'

After executing the above command, the status of the pd pod will change to ErrImagePull:

kubectl get pod my-cluster-pd-2 -n ${namespace}

NAME        READY   STATUS         RESTARTS   AGE
my-cluster-pd-2   0/1     ErrImagePull   0          57m

After pdFailoverPeriod (5m by default), a new PD Pod with the same name will be created and data will be moved to the new Pod.

One important difference between PD and TiKV failover is that if the failed PD pod comes back online, TiDB Operator automatically deletes the newly created node.