Automating Creation Of A Pub/Sub Topic And Subscription In GCP With Terraform And GitLab CI/CD

Automating Creation of a Pub/Sub Topic and Subscription in GCP with Terraform and GitLab CI/CD

Introduction: In the era of cloud computing, automating the deployment of cloud resources is essential for streamlined and efficient operations. In this blog post, we will explore how to automate the creation of a Pub/Sub topic and subscription in Google Cloud Platform (GCP) using Terraform. Additionally, we will utilize GitLab CI/CD to build a continuous integration and deployment pipeline that automates the entire process, providing a reliable and consistent approach to managing Pub/Sub resources.

Prerequisites: Before getting started, ensure you have the following prerequisites:

1. A Google Cloud Platform (GCP) account with sufficient permissions to create Pub/Sub topics and subscriptions.

2. A GitLab account with a repository set up to manage your Terraform code.

Repo Structure: For organized project management, let's follow this directory structure within our GitLab repository: GitLab-Repo

You can simply clone my public repository: GitLab-Repo

Terraform Configuration: Now let's delve into the details of the Terraform code for creating a Pub/Sub topic and subscription:

main.tf: The main.tf file contains the core Terraform configuration, defining the resources and their properties to be provisioned in the target cloud environment. Here, we set up a Google Cloud Pub/Sub topic and subscription.

resource "google_pubsub_topic" "example" {
  name = "sandbox-topic-gitlab"

  labels = {
    foo = "bar"
  }

  message_retention_duration = "86600s"
}


resource "google_pubsub_subscription" "example" {
  name  = "sandbox-subscription-gitlab"
  topic = google_pubsub_topic.example.name

  labels = {
    foo = "bar"
  }

  # 20 minutes
  message_retention_duration = "1200s"
  retain_acked_messages      = true

  ack_deadline_seconds = 20

  expiration_policy {
    ttl = "300000.5s"
  }
  retry_policy {
    minimum_backoff = "10s"
  }

  enable_message_ordering    = false
}

provider.tf: The provider.tf file specifies the configuration for the Terraform provider, defining the target cloud platform and its necessary details, such as the GCP project, region, and zone. It allows Terraform to interact with GCP and manage resources within the specified project.

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "4.58.0"
    }
  }
  backend "gcs" {
    bucket  = "your-backend-bucket" // Replace with your backend bucket name
    prefix  = "terraform/state"
  }
}

provider "google" {
  project = "your-project-id" // Replace with your project ID
  region  = "us-central1" // Replace with your desired region
  zone    = "us-central1-c" // Replace with your desired zone
}

GitLab CI/CD Configuration: The .gitlab-ci.yml file defines the CI/CD pipeline to automate the infrastructure deployment process. It specifies the stages, jobs, and corresponding scripts to perform tasks like validation, planning, applying, and destroying Terraform changes.

---
workflow:
  rules:
    - if: $CI_COMMIT_BRANCH != "main" && $CI_PIPELINE_SOURCE != "merge_request_event"
      when: never
    - when: always

variables:
  TF_DIR: ${CI_PROJECT_DIR}/terraform
  STATE_NAME: "gitlab-terraform-gcp-tf"

stages:
  - validate
  - plan
  - apply
  - destroy

image:
  name: hashicorp/terraform:light
  entrypoint: [""]
  
before_script:
  - terraform --version
  - cd ${TF_DIR}
  - terraform init -reconfigure

validate:
  stage: validate
  script:
    - terraform validate
  cache:
    key: ${CI_COMMIT_REF_NAME}
    paths:
    - ${TF_DIR}/.terraform
    policy: pull-push

plan:
  stage: plan
  script:
    - terraform plan 
  dependencies:
    - validate
  cache:
    key: ${CI_COMMIT_REF_NAME}
    paths:
    - ${TF_DIR}/.terraform
    policy: pull

apply:
  stage: apply
  script:
    - terraform apply  -auto-approve
  dependencies:
    - plan
  cache:
    key: ${CI_COMMIT_REF_NAME}
    paths:
    - ${TF_DIR}/.terraform
    policy: pull

destroy:
  stage: destroy
  script:
    - terraform destroy  -auto-approve
  dependencies:
    - plan
    - apply
  cache:
    key: ${CI_COMMIT_REF_NAME}
    paths:
    - ${TF_DIR}/.terraform
    policy: pull
  when: manual

Implementation Steps: Now, let's walk through the implementation steps to automate the creation of a Pub/Sub topic and subscription in GCP using Terraform and GitLab CI/CD:

1. Set up GitLab Repository: Create a new repository on GitLab or use an existing one to host your Terraform code. If you haven't already, clone the repository from the following link: GitLab-Repo

2. Configure GCP Provider: In the provider.tf file, configure the GCP provider by specifying your GCP project ID, region, and zone.

3. Set Secrets in GitLab: In your GitLab repository, navigate to Settings > CI/CD > Variables. Add a new variable named "GOOGLE_CREDENTIALS" and paste the contents of your Google Cloud service account key file into the value field. This securely provides the necessary credentials for Terraform to authenticate with GCP.

Note: Ensure to remove any white spaces in your token content before pasting it.

4. Run the Pipeline: Commit and push your Terraform code to the GitLab repository. This action will trigger the GitLab CI/CD pipeline. Monitor the pipeline execution in the CI/CD section of your repository to ensure it completes successfully.

5. Verify Resource Creation in GCP: After the pipeline is finished, verify the creation of the Pub/Sub topic and subscription in the Google Cloud Platform (GCP) Console. Ensure that the resources have been provisioned accurately.

Conclusion: In this blog post, we successfully automated the creation of a Pub/Sub topic and subscription in Google Cloud Platform using Terraform and GitLab CI/CD. By following the steps outlined above, you can now efficiently manage and automate your GCP Pub/Sub resources. Regularly updating your Terraform code and pipeline to reflect any changes in your infrastructure requirements ensures an agile and reliable infrastructure management approach. The combination of Terraform and GitLab CI/CD empowers you to automate infrastructure provisioning, improve consistency, and minimize errors in your cloud environment. Happy automating!

References: GitLab-Repo