Backup & Restore (BR)
- Objective: Learn to back up & restore a TiDB cluster on AWS (with Kubernetes)
- Prerequisites:
- Background knowledge of TiDB components
- Background knowledge of Kubernetes and TiDB Operator
- Background knowledge of BR
- AWS account
- TiDB cluster on AWS
- Optionality: Required
- Estimated time: 1 hour
This document describes how to perform physical backups of a TiDB cluster and use them to restore/recover a TiDB cluster.
Prepare
Optionality: Required
If you already created an S3 bucket and Kubernetes secrets as part of the Import & Export guide, you can skip this section.
Grant AWS Account Permissions
- Optionality: Required
Before you perform backup, AWS account permissions need to be granted to the Backup Custom Resource (CR) object. There are three methods to grant AWS account permissions:
- Grant permissions by importing AccessKey and SecretKey
- Grant permissions by associating IAM with Pod
- Grant permissions by associating IAM with ServiceAccount
In this doc, we use grant permissions by importing accesskey and secretkey
to grant AWS account permissions.
Note
Grant permissions by associating IAM with Pod or Grant permissions by associating IAM with ServiceAccount is recommanded in production enviroment.
Create S3 Bucket
If you don't have an S3 bucket for backup, you can create an S3 bucket in the same AWS region of your EKS cluster:
You can skip this section if you already have a S3 bucket to store backups.
Install RBAC
Download backup-rbac.yaml, and execute the following command to create the role-based access control (RBAC) resources in your namespace:
serviceaccount/tidb-backup-manager created
rolebinding.rbac.authorization.k8s.io/tidb-backup-manager created
Create Secrets
Create s3-secret
TiDB operator needs to access S3 when performing backup operations. To do that, you can create the s3-secret secret which stores the credential used to access S3:
kubectl create secret generic s3-secret --from-literal=access_key=${aws_access_key} --from-literal=secret_key=${aws_secret_key} --namespace=${namespace}
secret/s3-secret created
This s3-secret will be used in your Backup CR.
Verfify that the secret is properly created:
Create export-secret
TiDB operator needs to access TiDB when performing backup operations. To do that, you can create a secret which stores the password of the user account needed to access the TiDB cluster.
kubectl create secret generic export-secret --from-literal=password=${password} --namespace=${namespace}
secret/backup-secret created
Note
If there's no password for the user account, leave it blank in ${password}.
Verify Secrets
Verfify that the secret is properly created:
Ad-hoc Full Backup
- Optionality: Required
This section describes how to perform full backup. We use Backup Custom Resource (CR) to desbribe an ad-hoc full backup. TiDB Operator performs backup operation based on the specification in the Backup CR.
Configure Backup CR
The following is an example Backup CR to save as backup-aws-s3.yaml
:
You should replace values in {}
with specific variables in your envrioment and save in backup-aws-s3.yaml
.
namespace=<namespace>
cluster_name=<cluster_name>
region=<region>
bucket=<bucket>
prefix=<prefix>
cat > backup-aws-s3.yaml<<EOF
apiVersion: pingcap.com/v1alpha1
kind: Backup
metadata:
name: backup01
namespace: ${namespace}
spec:
backupType: full
br:
cluster: ${cluster_name}
sendCredToTikv: true
from:
host: ${cluster_name}-tidb
secretName: export-secret
s3:
provider: aws
secretName: s3-secret
region: ${region}
bucket: ${bucket}
prefix: ${prefix}
EOF
Perform Backup
You can perform an ad-hoc full backup using the following command:
backup.pingcap.com/backup01 created
Verify Backup
Check Backup Status
You can use the following command to check the backup status:
NAME BACKUPPATH BACKUPSIZE COMMITTS STARTED COMPLETED AGE
backup01 s3://bucket/prefix/ 1611872 416522333306486785 3m58s 3m55s 15m
Inspect Backup Log
You can use the following command to find the pod for the ad-hoc full backup:
You can then use the following command to check the backup progress:
...
I0508 02:56:48.029966 1 backup.go:92] [2020/05/08 02:56:48.029 +00:00] [INFO] [domain.go:607] ["domain closed"] ["take time"=2.586075ms]
I0508 02:56:48.030625 1 backup.go:92] [2020/05/08 02:56:48.030 +00:00] [INFO] [collector.go:203] ["Full backup Success summary: total backup ranges: 16, total success: 16, total failed: 0, total take(s): 1.98, total kv: 16000, total size(MB): 1.92, avg speed(MB/s): 0.97"] ["backup fast checksum"=1.561152ms] ["backup checksum"=21.194566ms] ["backup total regions"=16]
I0508 02:56:48.032898 1 backup.go:92]
I0508 02:56:48.033027 1 backup.go:107] Backup data for cluster anthony/backup01 successfully
I0508 02:56:48.040218 1 manager.go:207] reset cluster anthony/backup01 tikv_gc_life_time to 10m0s success
I0508 02:56:48.040241 1 manager.go:218] backup cluster anthony/backup01 data to s3://anthonybr/anthony_01/ success
I0508 02:56:48.136334 1 manager.go:232] Get size 1611872 for backup files in s3://anthonybr/anthony_01/ of cluster anthony/backup01 success
I0508 02:56:48.156090 1 manager.go:244] get cluster anthony/backup01 commitTs 416522333306486785 success
I0508 02:56:48.179889 1 backup_status_updater.go:66] Backup: [anthony/backup01] updated successfully
Check Backup Files
You can use the S3 console or aws s3
command to check backup files:
2020-05-17 17:06:03 0
2020-05-17 17:59:48 11598 1_100_25_3283d3a03adc06548749d97c17e127615132cbbf66b60da73a957847f19b62f7_write.sst
2020-05-17 17:59:48 187058 1_100_25_80992061af3e5194c3f28a5b79d486c5e9db2feda1afb3f84b4ca229ddce9932_write.sst
2020-05-17 17:59:48 187374 1_104_26_6d17617cf027713cbd7e33a2a124fe83bc2c35035a776d0c11e63551cbae7815_write.sst
2020-05-17 17:59:48 11487 1_104_26_a48ed62e9b0b2959f804d7850a940f2fc34e1382be1ab74d1518368e2512aba2_write.sst
2020-05-17 17:59:49 11544 1_108_27_2334a18b4455213018fb96ec3c92e951dcc4b10106e91896d882a5ee0d2dbea4_write.sst
2020-05-17 17:59:49 187315 1_108_27_4d4bf84bd30a3ae10b645ad9354c2c26cbc5ff2a95b338b93b820db95359617b_write.sst
2020-05-17 17:59:49 11551 1_116_29_96deb4607345137477d0eea19a6e7a200a37064f87c259d5fa8b08f3c52ed426_write.sst
2020-05-17 17:59:49 187206 1_116_29_9fb8afb3f7322e93f506f0d9d11e9b1569bc90b7c3779da9bb7e35137e8e6597_write.sst
2020-05-17 17:59:49 187222 1_2_29_393d3575060c0d616300c1199ef1c015784fdd3d13e950a6251007bbcbaf2c06_write.sst
2020-05-17 17:59:49 11640 1_2_29_e738a916dbc786a9aef2a69b15289ac934265a4fe851e1dc11140d2ac17b28e8_write.sst
2020-05-17 17:59:48 11556 1_92_23_c882bdf96118f3fcc64e32ac9040b3194ab67d910d1f5232b928416088211ba5_write.sst
2020-05-17 17:59:48 187312 1_92_23_f34f254bf742607feee767bcba259d92794f5f373e6fd415af086e4b42689491_write.sst
2020-05-17 17:59:48 187683 1_96_24_38e4059222579a22c8c937f6506dd8e2198a19c3d7fb4a245c0a4f804cd85adc_write.sst
2020-05-17 17:59:48 11487 1_96_24_675ff68620bc779c8c35ff210de4be67260ad904c9d18c17fed517a4fcf4226b_write.sst
2020-05-17 17:59:49 11564 5_112_28_20ea02d2b95115ebc0c5516aee14058da5e27a1f5bbc545fb831e6c3446fda82_write.sst
2020-05-17 17:59:49 187310 5_112_28_2550d727ffe3e408af41f541b134d5e064440c25988bbecfb665933d1195a45d_write.sst
2020-05-17 17:59:49 20770 backupmeta
Scheduled Full Backup
- Optionality: Optional
You can set up a backup policy to perform scheduled backups for a TiDB cluster, and set a backup retention policy. A scheduled full backup is described by a BackupSchedule CR object.
Configure BackupSchedule CR
The following is an example BackupSchedule CR. You should replace values including <>
with specific variables in your envrioment and save in backup-scheduler-aws-s3.yaml
.
namespace=<namespace>
cluster_name=<cluster_name>
aws_region=<aws_region>
bucket=<bucket>
prefix=<prefix>
cat > backup-scheduler-aws-s3.yaml<<EOF
apiVersion: pingcap.com/v1alpha1
kind: BackupSchedule
metadata:
name: demo-backup-schedule-s3
namespace: ${namespace}
spec:
#maxBackups: 5
#pause: true
maxReservedTime: "3h"
schedule: "*/2 * * * *"
backupTemplate:
backupType: full
br:
cluster: ${cluster_name}
clusterNamespace: ${namespace}
sendCredToTikv: true
from:
host: ${cluster_name}-tidb
secretName: export-secret
s3:
provider: aws
secretName: s3-secret
region: ${aws_region}
bucket: ${bucket}
prefix: ${prefix}
EOF
Perform Scheduled Backup
You can perform scheduled full backup using the following command:
backupschedule.pingcap.com/demo-backup-schedule-s3 created
Verify
Check Backup Status
You can use the following command to check the backup status:
NAME BACKUPPATH BACKUPSIZE COMMITTS STARTED COMPLETED AGE
backup02 s3://bucket/anthony_02/ 1611578 416550273190199297 5m1s 4m58s 5m12s
demo-backup-schedule-s3-2020-05-09t08-36-00 s3://bucket/anthony_sche/anthony-pd.anthony-2379-2020-05-09t08-36-00/ 1611578 416550320389226497 2m1s 118s 2m2s
demo-backup-schedule-s3-2020-05-09t08-38-00 0 2s
Inspect Backup Log
You can use the following command to find the pod for the scheduled full backup:
You can then use the following command to check the backup progress:
...
I0509 08:36:10.257977 1 backup.go:92] [2020/05/09 08:36:10.257 +00:00] [INFO] [client.go:149] ["save backup meta"] [path=s3://anthonybr/anthony_sche/anthony-pd.anthony-2379-2020-05-09t08-36-00] [jobs=0]
I0509 08:36:10.258477 1 backup.go:92] [2020/05/09 08:36:10.258 +00:00] [INFO] [progress.go:102] [Checksum] [progress=100.00%!](MISSING)
I0509 08:36:10.296059 1 backup.go:92] [2020/05/09 08:36:10.295 +00:00] [INFO] [ddl.go:407] ["[ddl] DDL closed"] [ID=061caa48-f309-422b-a53e-11623f55e1a8] ["take time"=973.143µs]
I0509 08:36:10.296130 1 backup.go:92] [2020/05/09 08:36:10.296 +00:00] [INFO] [ddl.go:301] ["[ddl] stop DDL"] [ID=061caa48-f309-422b-a53e-11623f55e1a8]
I0509 08:36:10.297872 1 backup.go:92] [2020/05/09 08:36:10.297 +00:00] [INFO] [domain.go:607] ["domain closed"] ["take time"=2.820476ms]
I0509 08:36:10.298207 1 backup.go:92] [2020/05/09 08:36:10.298 +00:00] [INFO] [collector.go:203] ["Full backup Success summary: total backup ranges: 16, total success: 16, total failed: 0, total take(s): 2.00, total kv: 16000, total size(MB): 1.92, avg speed(MB/s): 0.96"] ["backup checksum"=16.581592ms] ["backup fast checksum"=1.633146ms] ["backup total regions"=16]
I0509 08:36:10.300690 1 backup.go:92]
I0509 08:36:10.300730 1 backup.go:107] Backup data for cluster anthony/demo-backup-schedule-s3-2020-05-09t08-36-00 successfully
I0509 08:36:10.328351 1 manager.go:207] reset cluster anthony/demo-backup-schedule-s3-2020-05-09t08-36-00 tikv_gc_life_time to 10m0s success
I0509 08:36:10.328368 1 manager.go:218] backup cluster anthony/demo-backup-schedule-s3-2020-05-09t08-36-00 data to s3://anthonybr/anthony_sche/anthony-pd.anthony-2379-2020-05-09t08-36-00/ success
I0509 08:36:10.491981 1 manager.go:232] Get size 1611578 for backup files in s3://anthonybr/anthony_sche/anthony-pd.anthony-2379-2020-05-09t08-36-00/ of cluster anthony/demo-backup-schedule-s3-2020-05-09t08-36-00 success
I0509 08:36:10.520015 1 manager.go:244] get cluster anthony/demo-backup-schedule-s3-2020-05-09t08-36-00 commitTs 416550320389226497 success
I0509 08:36:10.541470 1 backup_status_updater.go:66] Backup: [anthony/demo-backup-schedule-s3-2020-05-09t08-36-00] updated successfully
Check Backup Files
You can use the S3 console or aws s3
command to check backup files:
Restore
- Optionality: Required
In this section, we will demonstrate how to use the backup created in previous sections to restore a database.
Cleanup
Note: that, BR only supports to restore to an empty cluster. To do that,you first need to login to the tidb cluster and drop the database
sbtest
:
MySQL [(none)]> drop database sbtest;
Query OK, 0 rows affected (0.31 sec)
Configure Restore CR
Similar to Backup, we use a Restore Custom Resource (CR) to describe a restore operation.
The following is an example Restore CR:
You should replace values including <>
with specific variables in your envrioment and save in restore-aws-s3.yaml
.
namespace=<namespace>
cluster_name=<cluster_name>
aws_region=<aws_region>
bucket=<bucket>
prefix=<prefix>
cat > restore-aws-s3.yaml<<EOF
apiVersion: pingcap.com/v1alpha1
kind: Restore
metadata:
name: demo-restore-s3
namespace: ${namespace}
spec:
br:
cluster: ${clustername}
clusterNamespace: ${namespace}
to:
host: ${cluster_name}-tidb
secretName: export-secret
s3:
provider: aws
secretName: s3-secret
region: ${aws_region}
bucket: ${bucket}
prefix: ${prefix}
EOF
Perform Restore
You can perform restore using the following command:
restore.pingcap.com/demo-restore-s3 created
Verify
Check Restore Status
You can use the following command to check the restore status:
NAME STARTED COMPLETED AGE
demo-restore-s3 43s 25s 49s
Inspect Restore Log
You can use the following command to check the retore process:
...
I0508 09:04:46.106204 1 restore.go:86] [2020/05/08 09:04:46.106 +00:00] [INFO] [collector.go:203] ["Full restore Success summary: total restore files: 16, total success: 16, total failed: 0, total take(s): 0.51, total kv: 16000, total size(MB): 1.92, avg speed(MB/s): 3.79"] ["split region"=110.963551ms] ["restore checksum"=19.994152ms] ["restore ranges"=16]
I0508 09:04:46.108742 1 restore.go:86]
I0508 09:04:46.108804 1 restore.go:101] Restore data for cluster anthony/demo-restore-s3 successfully
I0508 09:04:46.120192 1 manager.go:206] reset cluster anthony/demo-restore-s3 tikv_gc_life_time to 10m0s success
I0508 09:04:46.120219 1 manager.go:217] restore cluster anthony/demo-restore-s3 from succeed
I0508 09:04:46.143491 1 restore_status_updater.go:66] Restore: [anthony/demo-restore-s3] updated successfully
Check Database
Examine the state of the tables in TiDB after the restore completes:
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| INFORMATION_SCHEMA |
| METRICS_SCHEMA |
| PERFORMANCE_SCHEMA |
| mysql |
| sbtest |
| test |
+--------------------+
6 rows in set (0.00 sec)
mysql> use sbtest;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+------------------+
| Tables_in_sbtest |
+------------------+
| sbtest1 |
| sbtest2 |
| sbtest3 |
| sbtest4 |
| sbtest5 |
| sbtest6 |
| sbtest7 |
| sbtest8 |
+------------------+
8 rows in set (0.00 sec)
mysql> select count(*) from sbtest1;
+----------+
| count(*) |
+----------+
| 1000 |
+----------+
1 row in set (0.01 sec)
Comments
0 comments
Please sign in to leave a comment.