Best practices for upgrading TiDB
Date: 20 Jul 2022 Author: Daniël van Eeden Audience: PingCAP Status: Final
This article is meant to help you upgrade to a new major version. Some of these steps can be skipped for minor version upgrades.
This is written for TiDB Enterprise (on-premise), not all of this will apply to TiDB Cloud.
Make a backup with BR before starting the upgrade.
Read release notes
The release notes have details about changes that might affect the application after upgrading and also items that may impact the ability to do downgrades.
Use two clusters
Use TiCDC with two clusters.
Upgrade the secondary cluster, then switch roles and upgrade the primary cluster.
It is important to check the TiCDC Compatibility Notes to check for known issues with replicating between various versions.
Test on a test/acceptance system first
Use BR to restore a copy of the production system on acceptance/test and test the upgrade and upgrade procedure here first.
Be sure to do acceptance testing on any client application that uses this database as a backend.
Don't use new features directly
This is because most new features cause downgrades to be impossible. Details on this can be found in the release notes.
After running your client and cluster for a few weeks without compatibility incidents. You should be quite safe to begin using new features from the latest PingCAP release.
Know what downgrade options are available
Deployments with TiUP don't support downgrades. It may be possible to do this with many manual steps, but this is not something TiUP can do directly.
With kubernetes downgrades are possible, but not officially supported.
The only fully supported downgrade method is to dump and restore with Dumpling/Lightning.
Downgrading with BR backup/restore is not recommended. The BR documentation on compatibility has more details on what is known to cause issues with various version combinations.
Work with TiDB Support
Let us know via a Zendesk ticket that you plan to upgrade a production system. Depending on the contract there can be people standby to assist in case of issues and to review the plan beforehand.
Upgrade TiUP first
tiup update --self
tiup update --all
Upgrade TiDB Operator first
This is only when using Kubernetes
Make sure the cluster is healthy before you start
Wait for ongoing DDL to complete before starting the upgrade
Check for planned maintenance on AWS
Best to avoid having to deal with planned maintenance during the upgrade
Upgrade one cluster at a time
Wait at least a day between clusters to ensure any possible problems in the first cluster can be found and handled before starting with the next one.
Check the logs
Before and after the upgrade check the logs to see if there are any new warnings or errors that might need to be handled.
Prepare for applications to reconnect
Upgrading the TiDB nodes requires applications to reconnect. Most applications have auto-reconnect available and enabled.
Follow the docs
- Upgrade TiDB Using TiUP | PingCAP Docs
- Upgrade a TiDB Cluster in Kubernetes | PingCAP Docs
- Upgrade and After Upgrade FAQs | PingCAP Docs