- Gavin In The Cloud
- Posts
- Creating a BigQuery Dataset and a Table Using Terraform and GitHub Actions
Creating a BigQuery Dataset and a Table Using Terraform and GitHub Actions
Effortless BigQuery Setup: Terraform and GitHub Actions in Action
Creating a BigQuery Dataset and a Table Using Terraform and GitHub Actions
Introduction:
As a cloud engineer experienced in implementing solutions on GCP, you understand the significance of infrastructure as code (IAC) in automating and managing cloud resources. In this blog post, we will guide you through the process of creating a BigQuery dataset and a table in that dataset using Terraform and GitHub Actions. This automation ensures consistent data management and minimizes manual configuration efforts.

Prerequisites:
Before we delve into the technical details, ensure you have the following prerequisites in place:
An active Google Cloud Platform (GCP) account with appropriate permissions to create BigQuery resources.
A GitHub account with a repository set up to manage your Terraform code.
Repository Structure:
To maintain a structured project, follow a specific directory structure within your GitHub repository: GitHub-repo

You can clone my public repository for reference: GitHub-repo
Terraform Configuration:
Now, let's explore the details of our Terraform code that automates the creation of a BigQuery dataset and table:
main.tf: The main.tf
defines the Terraform configuration for creating a BigQuery dataset with specified attributes, including its ID, name, location, and table expiration settings, and it also defines a BigQuery table within that dataset with features like time partitioning and labels.
resource "google_bigquery_dataset" "dataset" {
dataset_id = "my_dataset"
friendly_name = "test"
description = "This is a dataset from Terraform script"
location = "US"
default_table_expiration_ms = 3600000
labels = {
env = "default"
}
}
resource "google_bigquery_table" "default" {
dataset_id = google_bigquery_dataset.dataset.dataset_id
table_id = "my-table"
time_partitioning {
type = "DAY"
}
labels = {
env = "default"
}
deletion_protection=false
}
provider.tf: The provider.tf
configures the Terraform provider for Google Cloud, specifying the required provider version and setting project, region, and zone parameters to define the target Google Cloud environment for resource provisioning.
terraform {
required_providers {
google = {
source = "hashicorp/google"
version = "4.58.0"
}
}
backend "gcs" {
bucket = "bigquery-demo-backend-bucket"
prefix = "terraform-state"
}
}
provider "google" {
project = "your-project"
region = "us-central1"
zone = "us-central1-c"
}
GitHub Actions Workflow: Our CI/CD pipeline is set up in the .github/workflows/terraform.yml file. It defines stages, jobs, and associated scripts to validate, plan, and apply Terraform changes.
name: "Deploy Big Query"
on:
push:
branches:
- main
jobs:
terraform:
name: "Terraform"
runs-on: ubuntu-latest
env:
GOOGLE_CREDENTIALS: ${{ secrets.GOOGLE_CREDENTIALS }}
defaults:
run:
working-directory: terraform
steps:
- name: Checkout
uses: actions/checkout@v2
- name: Setup Terraform
uses: hashicorp/setup-terraform@v1
with:
terraform_version: 1.0.1
terraform_wrapper: false
- name: Terraform Init
id: init
run: terraform init
- name: Terraform Format
id: fmt
run: terraform fmt
- name: Terraform Plan
id: plan
run: terraform plan
- name: Terraform Apply
id: apply
run: terraform apply -auto-approve
#- name: Terraform Destroy
#id: destroy
#run: terraform destroy -auto-approve
Implementation Steps:
With the configurations in place, here are the steps to implement this automated deployment:
Set Up GitHub Repository: Create a new GitHub repository or use an existing one to host your Terraform code. You can clone this repository as a reference: GitHub-repo
Configure GCP Provider: In the provider.tf file, configure the GCP provider with your project details, such as project ID, region, and zone.
Set Secrets in GitHub: In your GitHub repository, navigate to Settings > Secrets > New repository secret. Add a new secret named "GOOGLE_CREDENTIALS" and paste the contents of your Google Cloud service account key file into the value field. This securely provides the necessary credentials for Terraform to authenticate with GCP.

Note: Ensure that there are no whitespace characters in your token content before pasting it.
Automate with GitHub Actions: Commit and push your Terraform code to the GitHub repository. The GitHub Actions workflow will automatically trigger, executing the defined stages and jobs.
Verify Resources: Check the Google BigQuery console to confirm the successful creation of your dataset and table.
Conclusion:
In this blog post, we've demonstrated how to automate the creation of a BigQuery dataset and table using Terraform and GitHub Actions. By following the provided steps and configurations, you can efficiently manage and automate your GCP resources. This approach ensures consistency and reduces manual configuration efforts, making it a valuable addition to your cloud engineering toolkit. Happy automating!