tags: TiDB TiUP Google GCP Tutorial

Getting TiDB Up and Running on Google Cloud Compute (GCP) Instance - Part 1

Overview

The primary goal of this blog post series is to get a simple TiDB cluster up and running on Google GCP. This will allow you to try out TiDB and get familiar with the major components.

This is a two part post. The first part is setting up GCP so that we can run TiDB. The second part is setting up and running TiDB on the GCP environment that we created in the first part.

Here’s a Reference Architecture of what we will have at the end of the 2nd post.

Getting TiDB Up and Running on Google Cloud Compute (GCP) Instance - Part 1

Overview

Requirements

The Process

gcloud Account
Login with GCP User Account

In this first part, we will be focused on implementing Prerequisites Operations (see image below). Notice the two Swim Lanes (Local, GCP Instance). Each operation is run on the system associated with its specific Swim Lane

“Local” refers to your person/company computer. For me, my local computer is a MacBook Pro that is my daily computer. “GCP Instance”, is a virtual machine instance that we will create in Google GCP.

In each section, I will reference the image above with an arrow identifying the section we are working on.

Requirements

Basic understanding and access to Google GCP
Google GCP gcloud installed and configured on local computer

The Process

Validate Google gcloud Configuration on Local Computer

Alright, let’s start. On your local computer, you should have Google gcloud installed and configured.

On our local computer let’s take a look at gcloud configuration.

gcloud config list

Notice that I’m using a User Account; not a Service Account. Usually a service account will end with gserviceaccount.com.

In the following image, notice that I have compute configuration set for region and zone and you may also have this set. It shouldn’t matter if these values are set or not. In this getting started guide, when we use gcloud commands we will specify where the resource will live.

Create GCP Instance

Let’s create a GCP instance where we will run TiDB.

gcloud compute instances create tidb-vm \

--image-family=ubuntu-1804-lts \

--image-project=ubuntu-os-cloud \

--machine-type=e2-medium \

--boot-disk-size=200GB \

--boot-disk-type=pd-standard \

--boot-disk-device-name=tidb-vm \

--zone=us-west1-a

This may take a few moments to complete

Sanity Check

Let’s validate the instance is up and running

gcloud compute instances list

If you get an error and are not able to create an instance, you probably do not have valid permissions with the GCP account that gcloud is associated with.

SSH into New GCP Instance

Here we will use the gcloud ssh feature to access our newly created GCP instance.

gcloud compute ssh --zone "us-west1-a" "tidb-vm"

Notice that the server we SSH into has a prompt that references the tidb-vm instance.

GCP Instance - Change Service Account to User Account

gcloud Account

We should now be logged into the GCP instance tidb-vm

Let’s do something simple with gcloud that should validate our level of access, or the lack of access.

Let’s get a list of GCP instances .

gcloud compute instances lists

Notice that I received an error. This is because the account associated with gcloud is a Service Account that does not have permissions to get a list of instances. Even though you may not get an error, I recommend following the instructions below to change from a Services Account to your User Account.

A GCP Service Account is a special kind of account used by an application or a virtual machine (VM) instance, not a person. By default, GCP creates a Service Account and adds it to your project and is enabled on all instances.

Reference:

GCP Docs - Account Types
GCP Docs - Service Accounts

Let’s see what account we are running as. Here I run two commands that show similar information. Both of these commands provide similar information

gcloud config list

# or

gcloud auth list

Note the account email address and the domain. If it ends in compute@developer.gserviceaccount.com, you are running as a service account. In this getting started guide we want to execute gcloud commands with our User Account.

Let’s change the account that gcloud is associated with.

Login with GCP User Account

Here we will change the account that gcloud is associated with from a Service Account to our User Account.

It’s a multi-step process to change the gcloud account.

Reference: GCP Docs - Authorizing Cloud SDK tools

Let’s start the process by using the gcloud auth login command. This will initiate the process

gcloud auth login

You will see a long URL (see image above). Copy this url and paste it in a browser address bar.

In the browser, select the Google Account that has the permissions needed. For me, this is the same account I used when creating the GCP instance.

Click the “Allow” button

Click the copy button

Paste the verification code from the browser to the gcloud CLI.

Our User Account should now be associated with gcloud.

Sanity Check

Let’s confirm the account we are running as.

gcloud config list

# Or

gcloud auth list

Perfect, gcloud is now running under a User Account.

Let’s try again to get a list of machines.

gcloud compute instances lists

This doesn’t prove that we have all the access we need, but it’s a good baseline.

Create SSH Keys for TiUP

In Part 2 of this post, we will use TiUP to create TiDB clusters. TiUP uses SSH. We will need to configure SSH.

Think of TiUP as a package manager that makes it easier to manage different cluster components in the TiDB ecosystem. We will discuss TiUP more in Part 2 of this post.

Let’s see what’s in the .ssh directory

ls -al .ssh

There are no actual keys in this directory. We can use gcloud to create SSH key that TiUP will need.

Reference: Google Docs - gcloud compute config-ssh

When running the following command, you will be prompted to enter a passphrase. For simplicity, do not enter a passphrase when prompted. Just hit <return> to provide an empty passphrase.

gcloud compute config-ssh

Sanity Check

Let’s see what files were created.

ls -al .ssh/

Notice we now have 3 new files. The primary file we are interested in is the google_compute_engine, which is the private key that TiUP will use for SSH.

Open TiDB Ports on GCP Network

We will want to access TiDB resources (TiDB Dashboard, Grafana, Prometheus) from a browser on your local computer. To do this, we will need to open ports on the GCP network by creating a GCP Firewall Rule.

Below, we are going to create a firewall rule that will allow only our local computer to access TiDB resources.

Reference: Google Docs - VPC firewall rules overview

Let’s get our local computer public IP Address

Here’s a link that provides different ways of getting your local IP

How To Find My Public IP Address From Command Line On a Linux

We want our local computer IP address, NOT the GCP instance IP address. Therefore, run the following command on your local computer. For me, I’m running the following command on my MacBook. Notice in the image below, that the command prompt does not include tidb-vm

host myip.opendns.com resolver1.opendns.com

Notice my blurred-out IP address. We are going to use this IP Address below.

Let’s look at the current firewall rules

gcloud compute firewall-rules list

Notice, by default, that GCP already has some firewall rules. We will create a new firewall rule.

Here are the TiDB ports that we will make available to our local computer:

TiDB SQL Client: 4000
Grafana: 3000
Prometheus: 9090
TiDB Dashboard: 2379

PingCAP Docs - List of All TiDB Ports

Let’s create a firewall rule with our local IP Address.

In the code example below, for the parameter--source-ranges, replace <LOCAL IP> with your own local IP Address. Do keep the /32. This will limit it to your specific IP Address.

gcloud compute firewall-rules create access-from-home \

--allow tcp:4000,tcp:3000,tcp:9090,tcp:2379 \

--description="Allow traffic from my personal computer" \

--source-ranges="<LOCAL IP>/32"

I added port 2379 to the above command. It’s different than the image shows below.

In part 2, we will access TiDB from our local computer using these ports.

MySQL Client

TiDB is fully compatible with the MySQL 5.7 protocol and the common features and syntax of MySQL 5.7.

REFERENCE: PinCAP Docs - MySQL Compatibility

In Part 2, after we installed TiDB, we will access TiDB with a MySQL Client. Here we will install MySQL Client using apt-get

sudo apt-get update

sudo apt-get install mysql-client

Wrap Up

You should now have a GCP instance that you can use to install and run a simple TiDB environment.

Next is Part 2 where we install and run TiDB.

Getting TiDB Up and Running on Google Cloud Compute (GCP) Instance - Part 1

Overview

Requirements

The Process

Expand all

Back to top

Go to bottom

Select a repo

tags: TiDB TiUP Google GCP Tutorial

Getting TiDB Up and Running on Google Cloud Compute (GCP) Instance - Part 1

Overview

The primary goal of this blog post series is to get a simple TiDB cluster up and running on Google GCP. This will allow you to try out TiDB and get familiar with the major components.

This is a two part post. The first part is setting up GCP so that we can run TiDB. The second part is setting up and running TiDB on the GCP environment that we created in the first part.

Here’s a Reference Architecture of what we will have at the end of the 2nd post.

Table of Contents

In this first part, we will be focused on implementing Prerequisites Operations (see image below). Notice the two Swim Lanes (Local, GCP Instance). Each operation is run on the system associated with its specific Swim Lane

“Local” refers to your person/company computer. For me, my local computer is a MacBook Pro that is my daily computer. “GCP Instance”, is a virtual machine instance that we will create in Google GCP.

In each section, I will reference the image above with an arrow identifying the section we are working on.

Requirements

Basic understanding and access to Google GCP

Google GCP gcloud installed and configured on local computer

The Process

Validate Google gcloud Configuration on Local Computer

Alright, let’s start. On your local computer, you should have Google gcloud installed and configured.

On our local computer let’s take a look at gcloud configuration.

gcloud config list

Notice that I’m using a User Account; not a Service Account. Usually a service account will end with gserviceaccount.com.

In the following image, notice that I have compute configuration set for region and zone and you may also have this set. It shouldn’t matter if these values are set or not. In this getting started guide, when we use gcloud commands we will specify where the resource will live.

Create GCP Instance

Let’s create a GCP instance where we will run TiDB.

gcloud compute instances create tidb-vm \

--image-family=ubuntu-1804-lts \

--image-project=ubuntu-os-cloud \

--machine-type=e2-medium \

--boot-disk-size=200GB \

--boot-disk-type=pd-standard \

--boot-disk-device-name=tidb-vm \

--zone=us-west1-a

This may take a few moments to complete

Sanity Check

Let’s validate the instance is up and running

gcloud compute instances list

If you get an error and are not able to create an instance, you probably do not have valid permissions with the GCP account that gcloud is associated with.

SSH into New GCP Instance

Here we will use the gcloud ssh feature to access our newly created GCP instance.

gcloud compute ssh --zone "us-west1-a" "tidb-vm"

Notice that the server we SSH into has a prompt that references the tidb-vm instance.

GCP Instance - Change Service Account to User Account

gcloud Account

We should now be logged into the GCP instance tidb-vm

Let’s do something simple with gcloud that should validate our level of access, or the lack of access.

Let’s get a list of GCP instances .

gcloud compute instances lists

A GCP Service Account is a special kind of account used by an application or a virtual machine (VM) instance, not a person. By default, GCP creates a Service Account and adds it to your project and is enabled on all instances.

Reference:

Let’s see what account we are running as. Here I run two commands that show similar information. Both of these commands provide similar information

gcloud config list

# or

gcloud auth list

Note the account email address and the domain. If it ends in compute@developer.gserviceaccount.com, you are running as a service account. In this getting started guide we want to execute gcloud commands with our User Account.

Let’s change the account that gcloud is associated with.

Login with GCP User Account

Here we will change the account that gcloud is associated with from a Service Account to our User Account.

It’s a multi-step process to change the gcloud account.

Reference: GCP Docs - Authorizing Cloud SDK tools

Let’s start the process by using the gcloud auth login command. This will initiate the process

gcloud auth login

You will see a long URL (see image above). Copy this url and paste it in a browser address bar.

In the browser, select the Google Account that has the permissions needed. For me, this is the same account I used when creating the GCP instance.

Click the “Allow” button

Click the copy button

Paste the verification code from the browser to the gcloud CLI.

Our User Account should now be associated with gcloud.

Sanity Check

Let’s confirm the account we are running as.

gcloud config list

# Or

gcloud auth list

Perfect, gcloud is now running under a User Account.

Let’s try again to get a list of machines.

gcloud compute instances lists

This doesn’t prove that we have all the access we need, but it’s a good baseline.

Create SSH Keys for TiUP

In Part 2 of this post, we will use TiUP to create TiDB clusters. TiUP uses SSH. We will need to configure SSH.

Think of TiUP as a package manager that makes it easier to manage different cluster components in the TiDB ecosystem. We will discuss TiUP more in Part 2 of this post.

Let’s see what’s in the .ssh directory

ls -al .ssh