Skip to main content
Sign in

Backup & Recovery

Kolbe Kegel
  • Updated

Backup & Restore (BR)

  • Objective: Learn to back up & restore a TiDB cluster on AWS (with Kubernetes)
  • Prerequisites:
    • Background knowledge of TiDB components
    • Background knowledge of Kubernetes and TiDB Operator
    • Background knowledge of BR
    • AWS account
    • TiDB cluster on AWS
  • Optionality: Required
  • Estimated time: 1 hour

This document describes how to perform physical backups of a TiDB cluster and use them to restore/recover a TiDB cluster.

Prepare

Optionality: Required

If you already created an S3 bucket and Kubernetes secrets as part of the Import & Export guide, you can skip this section.

Grant AWS Account Permissions

  • Optionality: Required

Before you perform backup, AWS account permissions need to be granted to the Backup Custom Resource (CR) object. There are three methods to grant AWS account permissions:

In this doc, we use grant permissions by importing accesskey and secretkey to grant AWS account permissions.

Note

Grant permissions by associating IAM with Pod or Grant permissions by associating IAM with ServiceAccount is recommanded in production enviroment.

Create S3 Bucket

If you don't have an S3 bucket for backup, you can create an S3 bucket in the same AWS region of your EKS cluster:

create S3 bucket

You can skip this section if you already have a S3 bucket to store backups.

Install RBAC

Download backup-rbac.yaml, and execute the following command to create the role-based access control (RBAC) resources in your namespace:

kubectl apply -f backup-rbac.yaml -n ${namespace}
serviceaccount/tidb-backup-manager created
rolebinding.rbac.authorization.k8s.io/tidb-backup-manager created

Create Secrets

Create s3-secret

TiDB operator needs to access S3 when performing backup operations. To do that, you can create the s3-secret secret which stores the credential used to access S3:

kubectl create secret generic s3-secret --from-literal=access_key=${aws_access_key} --from-literal=secret_key=${aws_secret_key} --namespace=${namespace}
secret/s3-secret created

This s3-secret will be used in your Backup CR.

Verfify that the secret is properly created:

kubectl get secrets -n ${namespace}

Create export-secret

TiDB operator needs to access TiDB when performing backup operations. To do that, you can create a secret which stores the password of the user account needed to access the TiDB cluster.

kubectl create secret generic export-secret --from-literal=password=${password} --namespace=${namespace}
secret/backup-secret created

Note

If there's no password for the user account, leave it blank in ${password}.

Verify Secrets

Verfify that the secret is properly created:

kubectl get secrets -n ${namespace}

Ad-hoc Full Backup

  • Optionality: Required

This section describes how to perform full backup. We use Backup Custom Resource (CR) to desbribe an ad-hoc full backup. TiDB Operator performs backup operation based on the specification in the Backup CR.

Configure Backup CR

The following is an example Backup CR to save as backup-aws-s3.yaml:

You should replace values in {} with specific variables in your envrioment and save in backup-aws-s3.yaml.

namespace=<namespace>
cluster_name=<cluster_name>
region=<region>
bucket=<bucket>
prefix=<prefix>
cat > backup-aws-s3.yaml<<EOF
apiVersion: pingcap.com/v1alpha1
kind: Backup
metadata:
  name: backup01
  namespace: ${namespace}
spec:
  backupType: full
  br:
    cluster: ${cluster_name}
    sendCredToTikv: true
  from:
    host: ${cluster_name}-tidb
    secretName: export-secret
  s3:
    provider: aws
    secretName: s3-secret
    region: ${region}
    bucket: ${bucket}
    prefix: ${prefix}
EOF

Perform Backup

You can perform an ad-hoc full backup using the following command:

kubectl apply -f backup-aws-s3.yaml
backup.pingcap.com/backup01 created

Verify Backup

Check Backup Status

You can use the following command to check the backup status:

kubectl get bk -n ${namespace} -o wide
NAME       BACKUPPATH            BACKUPSIZE   COMMITTS             STARTED   COMPLETED   AGE
backup01   s3://bucket/prefix/   1611872      416522333306486785   3m58s     3m55s       15m

Inspect Backup Log

You can use the following command to find the pod for the ad-hoc full backup:

kubectl get po -n ${namespace}

You can then use the following command to check the backup progress:

kubectl logs -n ${namespace} -f ${backup_pod}
...
I0508 02:56:48.029966       1 backup.go:92] [2020/05/08 02:56:48.029 +00:00] [INFO] [domain.go:607] ["domain closed"] ["take time"=2.586075ms]
I0508 02:56:48.030625       1 backup.go:92] [2020/05/08 02:56:48.030 +00:00] [INFO] [collector.go:203] ["Full backup Success summary: total backup ranges: 16, total success: 16, total failed: 0, total take(s): 1.98, total kv: 16000, total size(MB): 1.92, avg speed(MB/s): 0.97"] ["backup fast checksum"=1.561152ms] ["backup checksum"=21.194566ms] ["backup total regions"=16]
I0508 02:56:48.032898       1 backup.go:92]
I0508 02:56:48.033027       1 backup.go:107] Backup data for cluster anthony/backup01 successfully
I0508 02:56:48.040218       1 manager.go:207] reset cluster anthony/backup01 tikv_gc_life_time to 10m0s success
I0508 02:56:48.040241       1 manager.go:218] backup cluster anthony/backup01 data to s3://anthonybr/anthony_01/ success
I0508 02:56:48.136334       1 manager.go:232] Get size 1611872 for backup files in s3://anthonybr/anthony_01/ of cluster anthony/backup01 success
I0508 02:56:48.156090       1 manager.go:244] get cluster anthony/backup01 commitTs 416522333306486785 success
I0508 02:56:48.179889       1 backup_status_updater.go:66] Backup: [anthony/backup01] updated successfully

Check Backup Files

You can use the S3 console or aws s3 command to check backup files:

aws s3 ls s3://${bucket}/${prefix}/
2020-05-17 17:06:03          0
2020-05-17 17:59:48      11598 1_100_25_3283d3a03adc06548749d97c17e127615132cbbf66b60da73a957847f19b62f7_write.sst
2020-05-17 17:59:48     187058 1_100_25_80992061af3e5194c3f28a5b79d486c5e9db2feda1afb3f84b4ca229ddce9932_write.sst
2020-05-17 17:59:48     187374 1_104_26_6d17617cf027713cbd7e33a2a124fe83bc2c35035a776d0c11e63551cbae7815_write.sst
2020-05-17 17:59:48      11487 1_104_26_a48ed62e9b0b2959f804d7850a940f2fc34e1382be1ab74d1518368e2512aba2_write.sst
2020-05-17 17:59:49      11544 1_108_27_2334a18b4455213018fb96ec3c92e951dcc4b10106e91896d882a5ee0d2dbea4_write.sst
2020-05-17 17:59:49     187315 1_108_27_4d4bf84bd30a3ae10b645ad9354c2c26cbc5ff2a95b338b93b820db95359617b_write.sst
2020-05-17 17:59:49      11551 1_116_29_96deb4607345137477d0eea19a6e7a200a37064f87c259d5fa8b08f3c52ed426_write.sst
2020-05-17 17:59:49     187206 1_116_29_9fb8afb3f7322e93f506f0d9d11e9b1569bc90b7c3779da9bb7e35137e8e6597_write.sst
2020-05-17 17:59:49     187222 1_2_29_393d3575060c0d616300c1199ef1c015784fdd3d13e950a6251007bbcbaf2c06_write.sst
2020-05-17 17:59:49      11640 1_2_29_e738a916dbc786a9aef2a69b15289ac934265a4fe851e1dc11140d2ac17b28e8_write.sst
2020-05-17 17:59:48      11556 1_92_23_c882bdf96118f3fcc64e32ac9040b3194ab67d910d1f5232b928416088211ba5_write.sst
2020-05-17 17:59:48     187312 1_92_23_f34f254bf742607feee767bcba259d92794f5f373e6fd415af086e4b42689491_write.sst
2020-05-17 17:59:48     187683 1_96_24_38e4059222579a22c8c937f6506dd8e2198a19c3d7fb4a245c0a4f804cd85adc_write.sst
2020-05-17 17:59:48      11487 1_96_24_675ff68620bc779c8c35ff210de4be67260ad904c9d18c17fed517a4fcf4226b_write.sst
2020-05-17 17:59:49      11564 5_112_28_20ea02d2b95115ebc0c5516aee14058da5e27a1f5bbc545fb831e6c3446fda82_write.sst
2020-05-17 17:59:49     187310 5_112_28_2550d727ffe3e408af41f541b134d5e064440c25988bbecfb665933d1195a45d_write.sst
2020-05-17 17:59:49      20770 backupmeta

Scheduled Full Backup

  • Optionality: Optional

You can set up a backup policy to perform scheduled backups for a TiDB cluster, and set a backup retention policy. A scheduled full backup is described by a BackupSchedule CR object.

Configure BackupSchedule CR

The following is an example BackupSchedule CR. You should replace values including <> with specific variables in your envrioment and save in backup-scheduler-aws-s3.yaml.

namespace=<namespace>
cluster_name=<cluster_name>
aws_region=<aws_region>
bucket=<bucket>
prefix=<prefix>
cat > backup-scheduler-aws-s3.yaml<<EOF
apiVersion: pingcap.com/v1alpha1
kind: BackupSchedule
metadata:
  name: demo-backup-schedule-s3
  namespace: ${namespace}
spec:
  #maxBackups: 5
  #pause: true
  maxReservedTime: "3h"
  schedule: "*/2 * * * *"
  backupTemplate:
    backupType: full
    br:
      cluster: ${cluster_name}
      clusterNamespace: ${namespace}
      sendCredToTikv: true
    from:
      host: ${cluster_name}-tidb
      secretName: export-secret
    s3:
      provider: aws
      secretName: s3-secret
      region: ${aws_region}
      bucket: ${bucket}
      prefix: ${prefix}
EOF

Perform Scheduled Backup

You can perform scheduled full backup using the following command:

kubectl apply -f backup-scheduler-aws-s3.yaml
backupschedule.pingcap.com/demo-backup-schedule-s3 created

Verify

Check Backup Status

You can use the following command to check the backup status:

kubectl get bk -n ${namespace} -o wide
NAME                                          BACKUPPATH                                                                 BACKUPSIZE   COMMITTS             STARTED   COMPLETED   AGE
backup02                                      s3://bucket/anthony_02/                                                 1611578      416550273190199297   5m1s      4m58s       5m12s
demo-backup-schedule-s3-2020-05-09t08-36-00   s3://bucket/anthony_sche/anthony-pd.anthony-2379-2020-05-09t08-36-00/   1611578      416550320389226497   2m1s      118s        2m2s
demo-backup-schedule-s3-2020-05-09t08-38-00                                                                              0                                                       2s

Inspect Backup Log

You can use the following command to find the pod for the scheduled full backup:

kubectl get pod -n ${namespace}

You can then use the following command to check the backup progress:

kubectl logs -n anthony -f ${backup_pod} 
...
I0509 08:36:10.257977       1 backup.go:92] [2020/05/09 08:36:10.257 +00:00] [INFO] [client.go:149] ["save backup meta"] [path=s3://anthonybr/anthony_sche/anthony-pd.anthony-2379-2020-05-09t08-36-00] [jobs=0]
I0509 08:36:10.258477       1 backup.go:92] [2020/05/09 08:36:10.258 +00:00] [INFO] [progress.go:102] [Checksum] [progress=100.00%!](MISSING)
I0509 08:36:10.296059       1 backup.go:92] [2020/05/09 08:36:10.295 +00:00] [INFO] [ddl.go:407] ["[ddl] DDL closed"] [ID=061caa48-f309-422b-a53e-11623f55e1a8] ["take time"=973.143µs]
I0509 08:36:10.296130       1 backup.go:92] [2020/05/09 08:36:10.296 +00:00] [INFO] [ddl.go:301] ["[ddl] stop DDL"] [ID=061caa48-f309-422b-a53e-11623f55e1a8]
I0509 08:36:10.297872       1 backup.go:92] [2020/05/09 08:36:10.297 +00:00] [INFO] [domain.go:607] ["domain closed"] ["take time"=2.820476ms]
I0509 08:36:10.298207       1 backup.go:92] [2020/05/09 08:36:10.298 +00:00] [INFO] [collector.go:203] ["Full backup Success summary: total backup ranges: 16, total success: 16, total failed: 0, total take(s): 2.00, total kv: 16000, total size(MB): 1.92, avg speed(MB/s): 0.96"] ["backup checksum"=16.581592ms] ["backup fast checksum"=1.633146ms] ["backup total regions"=16]
I0509 08:36:10.300690       1 backup.go:92]
I0509 08:36:10.300730       1 backup.go:107] Backup data for cluster anthony/demo-backup-schedule-s3-2020-05-09t08-36-00 successfully
I0509 08:36:10.328351       1 manager.go:207] reset cluster anthony/demo-backup-schedule-s3-2020-05-09t08-36-00 tikv_gc_life_time to 10m0s success
I0509 08:36:10.328368       1 manager.go:218] backup cluster anthony/demo-backup-schedule-s3-2020-05-09t08-36-00 data to s3://anthonybr/anthony_sche/anthony-pd.anthony-2379-2020-05-09t08-36-00/ success
I0509 08:36:10.491981       1 manager.go:232] Get size 1611578 for backup files in s3://anthonybr/anthony_sche/anthony-pd.anthony-2379-2020-05-09t08-36-00/ of cluster anthony/demo-backup-schedule-s3-2020-05-09t08-36-00 success
I0509 08:36:10.520015       1 manager.go:244] get cluster anthony/demo-backup-schedule-s3-2020-05-09t08-36-00 commitTs 416550320389226497 success
I0509 08:36:10.541470       1 backup_status_updater.go:66] Backup: [anthony/demo-backup-schedule-s3-2020-05-09t08-36-00] updated successfully

Check Backup Files

You can use the S3 console or aws s3 command to check backup files:

aws s3 ls s3://${bucket}/${prefix}/

Restore

  • Optionality: Required

In this section, we will demonstrate how to use the backup created in previous sections to restore a database.

Cleanup

Note: that, BR only supports to restore to an empty cluster. To do that，you first need to login to the tidb cluster and drop the database sbtest:

MySQL [(none)]> drop database sbtest;
Query OK, 0 rows affected (0.31 sec)

Configure Restore CR

Similar to Backup, we use a Restore Custom Resource (CR) to describe a restore operation.

The following is an example Restore CR:

You should replace values including <> with specific variables in your envrioment and save in restore-aws-s3.yaml.

namespace=<namespace>
cluster_name=<cluster_name>
aws_region=<aws_region>
bucket=<bucket>
prefix=<prefix>
cat > restore-aws-s3.yaml<<EOF
apiVersion: pingcap.com/v1alpha1
kind: Restore
metadata:
  name: demo-restore-s3
  namespace: ${namespace}
spec:
  br:
    cluster: ${clustername}
    clusterNamespace: ${namespace}

  to:
    host: ${cluster_name}-tidb
    secretName: export-secret
  s3:
    provider: aws
    secretName: s3-secret
    region: ${aws_region}
    bucket: ${bucket}
    prefix: ${prefix}
EOF

Perform Restore

You can perform restore using the following command:

kubectl apply -f resotre-aws-s3.yaml
restore.pingcap.com/demo-restore-s3 created

Verify

Check Restore Status

You can use the following command to check the restore status:

kubectl get rt -n ${namespace} -o wide
NAME              STARTED   COMPLETED   AGE
demo-restore-s3   43s       25s         49s

Inspect Restore Log

You can use the following command to check the retore process:

kubectl logs ${restore_pod} -n ${namespace}
...
I0508 09:04:46.106204       1 restore.go:86] [2020/05/08 09:04:46.106 +00:00] [INFO] [collector.go:203] ["Full restore Success summary: total restore files: 16, total success: 16, total failed: 0, total take(s): 0.51, total kv: 16000, total size(MB): 1.92, avg speed(MB/s): 3.79"] ["split region"=110.963551ms] ["restore checksum"=19.994152ms] ["restore ranges"=16]
I0508 09:04:46.108742       1 restore.go:86]
I0508 09:04:46.108804       1 restore.go:101] Restore data for cluster anthony/demo-restore-s3 successfully
I0508 09:04:46.120192       1 manager.go:206] reset cluster anthony/demo-restore-s3 tikv_gc_life_time to 10m0s success
I0508 09:04:46.120219       1 manager.go:217] restore cluster anthony/demo-restore-s3 from  succeed
I0508 09:04:46.143491       1 restore_status_updater.go:66] Restore: [anthony/demo-restore-s3] updated successfully

Check Database

Examine the state of the tables in TiDB after the restore completes:

show databases;
use sbtest;
show tables;
mysql> show databases; 
+--------------------+
| Database           |
+--------------------+
| INFORMATION_SCHEMA |
| METRICS_SCHEMA     |
| PERFORMANCE_SCHEMA |
| mysql              |
| sbtest             |
| test               |
+--------------------+
6 rows in set (0.00 sec)

mysql> use sbtest; 
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
+------------------+
| Tables_in_sbtest |
+------------------+
| sbtest1          |
| sbtest2          |
| sbtest3          |
| sbtest4          |
| sbtest5          |
| sbtest6          |
| sbtest7          |
| sbtest8          |
+------------------+
8 rows in set (0.00 sec)

mysql> select count(*) from sbtest1;
+----------+
| count(*) |
+----------+
|     1000 |
+----------+
1 row in set (0.01 sec)

Related articles

Comments

0 comments

Please sign in to leave a comment.