Best practices for upgrading TiDB

Date: 20 Jul 2022 Author: Daniël van Eeden Audience: PingCAP Status: Final

This article is meant to help you upgrade to a new major version. Some of these steps can be skipped for minor version upgrades.

This is written for TiDB Enterprise (on-premise), not all of this will apply to TiDB Cloud.

Backups

Make a backup with BR before starting the upgrade.

Read release notes

The release notes have details about changes that might affect the application after upgrading and also items that may impact the ability to do downgrades.

Use two clusters

Use TiCDC with two clusters.

Upgrade the secondary cluster, then switch roles and upgrade the primary cluster.

It is important to check the TiCDC Compatibility Notes to check for known issues with replicating between various versions.

Docs: Replicate data between primary and secondary clusters | PingCAP Docs

Test on a test/acceptance system first

Use BR to restore a copy of the production system on acceptance/test and test the upgrade and upgrade procedure here first.

Be sure to do acceptance testing on any client application that uses this database as a backend.

Don't use new features directly

This is because most new features cause downgrades to be impossible. Details on this can be found in the release notes.

After running your client and cluster for a few weeks without compatibility incidents. You should be quite safe to begin using new features from the latest PingCAP release.

Know what downgrade options are available

Deployments with TiUP don't support downgrades. It may be possible to do this with many manual steps, but this is not something TiUP can do directly.

With kubernetes downgrades are possible, but not officially supported.

The only fully supported downgrade method is to dump and restore with Dumpling/Lightning.

Downgrading with BR backup/restore is not recommended. The BR documentation on compatibility has more details on what is known to cause issues with various version combinations.

Work with TiDB Support

Let us know via a Zendesk ticket that you plan to upgrade a production system. Depending on the contract there can be people standby to assist in case of issues and to review the plan beforehand.

Upgrade TiUP first

tiup update --self

tiup update --all

Upgrade TiDB Operator first

This is only when using Kubernetes

Make sure the cluster is healthy before you start

Wait for ongoing DDL to complete before starting the upgrade

Check for planned maintenance on AWS

Best to avoid having to deal with planned maintenance during the upgrade

Upgrade one cluster at a time

Wait at least a day between clusters to ensure any possible problems in the first cluster can be found and handled before starting with the next one.

Check the logs

Before and after the upgrade check the logs to see if there are any new warnings or errors that might need to be handled.

Prepare for applications to reconnect

Upgrading the TiDB nodes requires applications to reconnect. Most applications have auto-reconnect available and enabled.

Follow the docs