Deploy an EKS Cluster

• Objective: Learn to deploy a Kubernetes cluster on AWS (with EKS)
• Prerequisites:
  • AWS account
• Optionality: Required
• Estimated time: 40min

Prepare tools

• Optionality: Required

To deploy a Kubernetes Cluster on AWS EKS, some tools are required to interact with AWS and Kubernetes. The tools can be installed on any host that can access the AWS server, and subsequent operations will be performed on this host.

Note

You can continue to manually download and install tools, or use the all-in-one prepared Docker image directly. If you do choose to use the Docker image, note that port forwarding for web-based services is more complex, because ports are forwarded from the Docker container rather than your local machine.

Manually download tools

Below is the prerequisites list. Please follow the instruction links to install them.

Verify tools

Note: Please pay special attention to helm's version; helm 3.0 is not supported.

$ aws --version
aws-cli/2.0.7 Python/3.7.3 Linux/4.15.0-66-generic botocore/2.0.0dev11

$ terraform --version
Terraform v0.12.24

$ kubectl version --client
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-08T17:38:50Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

$ helm version --client
Client: &version.Version{SemVer:"v2.16.3", GitCommit:"1ee0254c86d4ed6887327dabed7aa7da29d7eb0d", GitTreeState:"clean"}

$ jq --version
jq-1.5-1-a5b5cbe

$ aws-iam-authenticator version
{"Version":"v0.5.0","Commit":"1cfe2a90f68381eacd7b6dcfa2bf689e76eb8b4b"}

Troubleshooting

bash: /usr/local/bin/helm: cannot execute binary file

Please check whether the correct platform was selected when downloading tools.

Use all-in-one prepared Docker image

If you have already downloaded and installed the tools, skip ahead to Create Kubernetes Cluster.

Run the following command to enter the container environment that contains all required tools:

docker run -it disksing/tidb-eks-tools:latest

Create Kubernetes Cluster

• Optionality: Required

Configure AWS credentials

Use aws configure to setup access keys.

Note: The access key must have at least these permissions:

• Create VPC
• Create EBS volumes
• Create EC2 instance
• Create roles

Commands

Note: the following keys are examples, please replace with your own keys.

aws configure

AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json

Verify Keys

aws configure list

      Name                    Value             Type    Location
      ----                    -----             ----    --------
   profile                <not set>             None    None
access_key     ******************** shared-credentials-file
secret_key     ******************** shared-credentials-file
    region                us-west-2      config-file    ~/.aws/config

Clone the tidb-operator repo

The tidb-operator project provides utilities and configurations for creating an EKS cluster that is suitable for deployment of TiDB components.

Commands

git clone --branch release-1.1 --depth=1 https://github.com/pingcap/tidb-operator.git
cd tidb-operator/deploy/aws && ls

Verify clone

README.md  aws-tutorial.tfvars  clusters.tf  default-cluster.yaml  demo.tfvars  main.tf  manifests  outputs.tf  prod.tfvars  variables.tf  versions.tf

Install Terraform Dependencies

Terraform needs to install some necessary components to work.

Commands

terraform init

...
Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Configure EKS Cluster

Some Terraform configuration files are pre-defined in the aws directory. You can modify the configuration file to customize deploy region, cluster size, instance type, etc.

Since applying for resources is a time-consuming operation (about 20+ minutes), we recommend that you first create a cluster using the default configuration we provide, and then during the creation process you can take your time to learn how to customize the configurations.

Note: you can create EKS cluster with default configuration files directly.

Configure AWS region

You can customize AWS region by editing the variables.tf file. All avaliable values are listed in comments. The default value is us-west-2.

variable "region" {
  description = "AWS region"
  # supported regions:
  # US: us-east-1, us-east-2, us-west-2
  # Asia Pacific: ap-south-1, ap-northeast-2, ap-southeast-1, ap-southeast-2, ap-northeast-1
  # Europe: eu-central-1, eu-west-1, eu-west-2, eu-west-3, eu-north-1
  default = "us-west-2"
}

Configure cluster name

You can customize EKS cluster name or TiDB cluster name by editing the variables.tf file.

Note: Be aware, if you change the cluster name, in the following steps, some commands may need to be adjusted accordingly.

variable "eks_name" {
  description = "Name of the EKS cluster. Also used as a prefix in names of related resources."
  default     = "my-cluster"
}
variable "default_cluster_name" {
  default = "my-cluster"
}

Configure cluster size

You can customize cluster size by editing the variables.tf file. By default, the cluster contains 3 PDs, 3 TiKVs and 2 TiDBs.

Note: if you change the cluster size, in the following steps, the output of some commands and cluster status may be quite different from what is shown.

variable "default_cluster_pd_count" {
  default = 3
}
variable "default_cluster_tikv_count" {
  default = 3
}
variable "default_cluster_tidb_count" {
  default = 2
}

Configure instance type

You can customize instance type by editing the variables.tf file. The default confirations are listed below. The default configuration is suitable for basic functional evaluation, not suitable for performance tests.

AWS EC2 instance pricing contains more detailed information about instance types and their prices.

variable "default_cluster_pd_instance_type" {
 default = "c5d.large"
}
 
variable "default_cluster_tikv_instance_type" {
 default = "c5d.large"
}
 
variable "default_cluster_tidb_instance_type" {
 default = "c4.large"
}
 
variable "default_cluster_monitor_instance_type" {
 default = "c5.large"
}

Create EKS Cluster

We will use Terraform to create AWS resources and create EKS clusters based on the information provided by the configuration file.

Commands

terraform apply

Terraform does not directly perform operations, but first outputs the operation plan in the form of a diff. At the end of the command line output, there will be a line summarizing the operations to be performed and prompting you to confirm:

Plan: 66 to add, 0 to change, 0 to destroy.
...
Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

To proceed, you need to type yes and hit Enter. The running process will last more than 20 minutes, and a status message will be output every 10 seconds during the running process, which includes the total time currently used:

module.tidb-operator.null_resource.setup-env: Still creating... [1m10s elapsed]
module.tidb-operator.null_resource.setup-env: Still creating... [1m20s elapsed]
module.tidb-operator.null_resource.setup-env: Still creating... [1m30s elapsed]
module.tidb-operator.null_resource.setup-env: Still creating... [1m40s elapsed]
module.tidb-operator.null_resource.setup-env: Still creating... [1m50s elapsed]
module.tidb-operator.null_resource.setup-env: Still creating... [2m0s elapsed]

Note: If you are creating using the default configration, you can take this time to learn about how to Configure EKS Cluster.

When it is finally completed, you will see command line output like this:

Apply complete! Resources: 66 added, 0 changed, 0 destroyed.

Outputs:

bastion_ip = [
  "52.25.125.150",
]
default-cluster_monitor-dns = not_created
default-cluster_tidb-dns = not_created
eks_endpoint = https://F989422954FCA9A08505C4BBE98FA194.sk1.us-west-2.eks.amazonaws.com
eks_version = 1.15
kubeconfig_filename = credentials/kubeconfig_my-cluster
region = us-west-2

The output contains the information needed to connect the EKS cluster. The bastion_ip is the IP address of an EC2 server that can access the TiDB server. You can use terraform output to display the output later any time you want to reference it.

Troubleshooting

Error: No valid credential sources found for AWS Provider

You need to configure AWS credentials before creating the cluster.

Error: Provider produced inconsistent final plan

This is a known issue with Terraform. Please execute terraform apply again to resolve the situation.

Error: deployments.apps "tiller-deploy" not found

The installed Helm version is too new. Currently the Terraform scripts for TiDB Operator do not support helm 3. Please refer to Manually download tools to install the correct version.

terraform apply command fails

In some cases, a failure of terraform apply can be resolved by simply running it again.