Create GKE Cluster with Terraform/Opentofu 0x03
Part 3 of this post
Table of Contents
To make this really modular let’s create environemnt for Dev and Prod.
Let’s assign standard gke to dev and autopilot gke to prod.
This approach keeps our infrastructure:
- consistent
- reusable
- environment-aware
- easy to scale
Providers
This is present in both environment.
provider.tf
1terraform {
2 required_providers {
3 google = {
4 source = "hashicorp/google"
5 version = "7.26.0"
6 }
7 kubernetes = {
8 source = "hashicorp/kubernetes"
9 version = "~> 2.0"
10 }
11
12 helm = {
13 source = "hashicorp/helm"
14 version = "~> 2.10" # or latest
15 }
16 }
17}
18
19provider "google" {
20 credentials = file(var.credentials_file)
21 project = var.project_id
22 region = var.region
23}
24
25data "google_client_config" "default" {}
26
27provider "kubernetes" {
28 host = "https://${module.gke.endpoint}"
29 token = data.google_client_config.default.access_token
30
31 cluster_ca_certificate = base64decode(
32 module.gke.ca_certificate
33 )
34}
35
36provider "helm" {
37 kubernetes {
38 host = "https://${module.gke.endpoint}"
39
40 token = data.google_client_config.default.access_token
41
42 cluster_ca_certificate = base64decode(
43 module.gke.ca_certificate
44 )
45 }
46}
Development Environment
The dev environment is designed for flexibility and cost efficiency.
It uses a Standard GKE cluster, where you have full control over nodes.
env/dev/main.tf
1# -------------------------------
2# IAM
3# -------------------------------
4module "api" {
5 source = "../../modules/iam/api"
6 project_id = var.project_id
7}
8
9module "service-account" {
10 source = "../../modules/iam/service-account"
11 node_sa = var.node_sa
12 project_id = var.project_id
13}
14
15# -------------------------------
16# Network
17# -------------------------------
18module "network" {
19 source = "../../modules/network"
20 region = var.region
21
22}
23
24# -------------------------------
25# Cluster
26# -------------------------------It uses GKE Autopilot, where Google manages the infrastructure.
27module "gke" {
28 source = "../../modules/gke/standard"
29
30 project_id = var.project_id
31 region = var.region
32 zone = var.zone
33
34 cluster_name = var.cluster_name
35 initial_node_count = var.initial_node_count
36
37 network = module.network.network
38 subnetwork = module.network.subnetwork
39}
40
41# -------------------------------
42# Node Pool
43# -------------------------------
44module "nodepool" {
45 source = "../../modules/nodepool"
46
47 project_id = var.project_id
48 region = var.region
49 cluster_id = module.gke.cluster_id
50
51 node_sa = module.service-account.node_sa_email
52
53 min_node = var.min_node
54 max_node = var.max_node
55
56 disk_size_gb = var.disk_size_gb
57 machine_type = var.machine_type
58 disk_type = var.disk_type
59
60}
61
62# -------------------------------
63# Storage
64# -------------------------------
65module "storage" {
66 source = "../../modules/storage"
67
68 balanced_sc_name = var.balanced_sc_name
69 ssd_sc_name = var.ssd_sc_name
70
71
72 depends_on = [module.gke, module.nodepool]
73}
74
75# -------------------------------
76# Ingress-nginx
77# -------------------------------
78module "ingress-nginx" {
79 source = "../../modules/addons/ingress-nginx"
80
81 depends_on = [module.gke, module.nodepool]
82}
83
84# -------------------------------
85# Cert-manager
86# -------------------------------
87module "cert_manager" {
88 source = "../../modules/addons/cert-manager"
89
90 depends_on = [module.gke, module.nodepool, module.ingress-nginx]
91}
env/dev/terraform.tfvars
1project_id = "project-12345"
2credentials_file = "/home/username/.config/gcp/service-account.json"
3region = "us-east1"
4zone = "us-east1-c"
5
6service_account = "tofu-sa@project-123456.iam.gserviceaccount.com"
7
8node_sa = "gke-node-sa"
9cluster_name = "gke-cluster"
10
11initial_node_count = 1
12
13disk_size_gb = 20
14machine_type = "e2-small"
15disk_type = "pd-balanced"
16
17min_node = 1
18max_node = 5
19
20balanced_sc_name = "balanced-sc"
21ssd_sc_name = "ssd-sc"
Production Environment
Production is designed for stability and minimal operations overhead.
It uses GKE Autopilot, where Google manages the infrastructure.
env/dev/main.tf
1# -------------------------------
2# IAM
3# -------------------------------
4module "api" {
5 source = "../../modules/iam/api"
6 project_id = var.project_id
7}
8
9# -------------------------------
10# Network
11# -------------------------------
12module "network" {
13 source = "../../modules/network"
14 region = var.region
15
16}
17
18# -------------------------------
19# Cluster
20# -------------------------------
21module "gke" {
22 source = "../../modules/gke/autopilot"
23
24 project_id = var.project_id
25 region = var.region
26 zone = var.zone
27
28 cluster_name = var.cluster_name
29
30 network = module.network.network
31 subnetwork = module.network.subnetwork
32}
33
34# -------------------------------
35# Storage
36# -------------------------------
37module "storage" {
38 source = "../..//modules/storage"
39
40 balanced_sc_name = var.balanced_sc_name
41 ssd_sc_name = var.ssd_sc_name
42
43
44 depends_on = [module.gke]
45}
46
47# -------------------------------
48# Ingress-nginx
49# -------------------------------
50module "ingress-nginx" {
51 source = "../../modules/addons/ingress-nginx"
52
53 depends_on = [module.gke]
54}
55
56# -------------------------------
57# Cert-manager
58# -------------------------------
59module "cert_manager" {
60 source = "../../modules/addons/cert-manager"
61
62 depends_on = [module.gke, module.ingress-nginx]
63}
env/prod/terraform.tfvars
1project_id = "project-123456"
2credentials_file = "/home/username/.config/gcp/service-account.json"
3region = "us-east1"
4zone = "us-east1-c"
5
6service_account = "tofu-sa@project-123456.iam.gserviceaccount.com"
7
8cluster_name = "gke-cluster"
9
10balanced_sc_name = "balanced-sc"
11ssd_sc_name = "ssd-sc"
Deploy
Navigate to either the environment.
- env/dev
- env/prod
1tofu plan
2tofu apply
Kubeconfig
Get config using this command:
1gcloud container clusters get-credentials <gke-cluster-name> --region <region> --project <project-id>
Verify
Verify using kubectl command.
1kubectl get pods -A
2NAMESPACE NAME READY STATUS RESTARTS AGE
3cert-manager cert-manager-cainjector-7fc5bf87cd-bh96c 1/1 Running 0 3m19s
4cert-manager cert-manager-fb659bc9b-9fwg6 1/1 Running 0 3m19s
5cert-manager cert-manager-webhook-55b4ff9fbf-pcngn 1/1 Running 0 3m19s
6gke-managed-cim kube-state-metrics-0 2/2 Running 0 13m
7gmp-system collector-725mp 2/2 Running 0 7m31s
8gmp-system collector-cpwlq 2/2 Running 0 5m51s
9gmp-system gmp-operator-68d4ff44c6-whc2g 1/1 Running 0 12m
10ingress-nginx ingress-nginx-controller-6c7cd85885-zhf2k 1/1 Running 0 6m35s
11kube-system event-exporter-gke-77766c7db8-gw9kh 2/2 Running 1 (6m20s ago) 13m
12kube-system fluentbit-gke-6td9k 3/3 Running 0 5m51s
13kube-system fluentbit-gke-dkj46 3/3 Running 0 7m31s
14kube-system gke-metadata-server-gmtjs 1/1 Running 0 7m31s
15kube-system gke-metadata-server-j7b8k 1/1 Running 0 5m51s
16kube-system gke-metrics-agent-hn6dz 3/3 Running 0 5m51s
17kube-system gke-metrics-agent-kswdw 3/3 Running 0 7m31s
18kube-system ip-masq-agent-7x4px 1/1 Running 0 7m31s
19kube-system ip-masq-agent-8ddst 1/1 Running 0 5m51s
20kube-system konnectivity-agent-autoscaler-6f99bf84bc-b45zf 1/1 Running 0 12m
21kube-system konnectivity-agent-cc694844f-6crvt 2/2 Running 0 5m41s
22kube-system konnectivity-agent-cc694844f-ssn8g 2/2 Running 0 12m
23kube-system kube-dns-5865677fdd-mqb4t 4/4 Running 0 5m41s
24kube-system kube-dns-5865677fdd-z9kzr 4/4 Running 0 13m
25kube-system kube-dns-autoscaler-6c49bf7f57-pqgpk 1/1 Running 0 12m
26kube-system kube-proxy-gke-gke-cluster-gke-node-c8d441f8-plbd 1/1 Running 0 7m31s
27kube-system kube-proxy-gke-gke-cluster-gke-node-d17708a2-km2d 1/1 Running 0 5m51s
28kube-system metrics-server-v1.35.1-7f6b8db48b-8wvrq 1/1 Running 1 (4m13s ago) 12m
29kube-system netd-7b89w 3/3 Running 0 5m51s
30kube-system netd-8hs6w 3/3 Running 0 7m31s
31kube-system node-local-dns-v9f44 2/2 Running 0 5m51s
32kube-system node-local-dns-ztzqc 2/2 Running 0 7m31s
33kube-system pdcsi-node-mc2gf 3/3 Running 0 5m51s
34kube-system pdcsi-node-mz4v4 3/3 Running 0 7m31s
35
36kubectl get nodes -A
37NAME STATUS ROLES AGE VERSION
38gke-gke-cluster-gke-node-c8d441f8-plbd Ready <none> 7m38s v1.35.1-gke.1396002
39gke-gke-cluster-gke-node-d17708a2-km2d Ready <none> 5m59s v1.35.1-gke.1396002