Provision CelerData Cloud BYOC on GCP

Step by step deployment of a CelerData Cloud BYOC cluster on GCP:

• Preparations
• Configure providers
• Configure GCP objects
• Describe infrastructure
• Apply configurations

Read this article before you start a Terraform configuration for your cluster deployment on GCP.

Preparations

Before using the CelerData Cloud BYOC provider to create infrastructure at the GCP account level for the first time, you must complete the following preparations:

GCP prerequisites

1. Have an GCP service account with administrative privileges.
2. Install the GCP CLI. For more information, see How to install the GCP CLI.

CelerData prerequisites

Obtain the credentials to authenticate into the CelerData Cloud BYOC platform. For details, see Authentication.

Terraform prerequisites

1. Install Terraform in your terminal.

2. Have a Terraform project. In your terminal, create an empty directory (for example, terraform) and then switch to it. (Each separate set of Terraform configuration files must be in its own directory, which is called a Terraform project.)

Configure providers

This section assumes that you have completed the preparations.

Create a .tf file (for example, main.tf) in your Terraform project. Then, add the following code snippet to the .tf file:

terraform {
  required_providers {
    celerdatabyoc = {
      source  = "CelerData/celerdatabyoc"
      version = "<provider_version>"
    }
    google = {
      source = "hashicorp/google"
      version = "6.37.0"
    }
  }
}

provider "celerdatabyoc" {
  client_id = "<client_id>"
  client_secret = "<client_secret>"
}

provider "google" {
  project     = "<project_id>"
  region      = "<region>"
}

The parameters you need to specify are as follows:

• provider_version: Enter the CelerData provider version of your choice. We recommend that you select the latest provider version, for example version = "1.0.3". You can view the provider versions offered by CelerData Cloud BYOC from the CelerData Cloud BYOC provider page.
• client_id and client_secret: Enter the Client ID and Secret of your application key. See For CelerData.
• project_id: Enter your GCP cloud project ID.
• region: Enter the ID of the GCP cloud region in which you want your CelerData cluster to run. See Supported cloud platforms and regions.

Configure GCP objects

The deployment on GCP requires creating a Data Credential, Deployment Credential, and Network Credential in CelerData Cloud. For detailed steps, please refer to: Cloud Setting For GCP

• Create Deployment Credential
• Create Data Credential
• Create Network Credential

Here is an example written based on the document:

provider "google" {
  project     = "<gcp_project_id>"
  region      = "us-central1"
}

resource "google_project_service" "project" {
  for_each = toset([
    "analyticshub.googleapis.com",
    "bigquery.googleapis.com",
    "bigqueryconnection.googleapis.com",
    "bigquerydatapolicy.googleapis.com",
    "bigquerymigration.googleapis.com",
    "bigqueryreservation.googleapis.com",
    "bigquerystorage.googleapis.com",
    "compute.googleapis.com",
    "cloudapis.googleapis.com",
    "logging.googleapis.com",
    "iamcredentials.googleapis.com",
    "dataplex.googleapis.com",
    "datastore.googleapis.com",
    "dns.googleapis.com",
    "monitoring.googleapis.com",
    "oslogin.googleapis.com",
    "sqladmin.googleapis.com",
    "storage.googleapis.com",
    "cloudtrace.googleapis.com",
    "dataform.googleapis.com",
    "storage-api.googleapis.com",
    "privilegedaccessmanager.googleapis.com",
    "servicedirectory.googleapis.com",
    "servicemanagement.googleapis.com",
    "servicenetworking.googleapis.com",
    "serviceusage.googleapis.com",
  ])
  service            = each.key
  disable_on_destroy = false
}

resource "google_storage_bucket" "storage-bucket" {
  name     = "tf-test-bucket-${local.gcp_project_id}"
  location = us-central1
  storage_class = "STANDARD"
  uniform_bucket_level_access = false
  public_access_prevention    = "enforced"

  force_destroy = false

  labels = {
    environment = "prod"
  }

  lifecycle {
    ignore_changes = [
      cors
    ]
  }
}

resource "google_service_account" "storage-sa" {
  account_id   = "tf-test-storage-sa"
  display_name = "Terraform Test Service Account"
}

resource "google_project_iam_custom_role" "storage-custom-role" {
  role_id     = "tfteststoragerole"
  title       = "CelerData Compute Engine Storage Role"
  description = "Custom role with only necessary permissions for CelerData"
  project     = "<gcp_project_id>"

  permissions = [
    "iam.serviceAccounts.signBlob",
    "storage.buckets.get",
    "storage.buckets.update",
    "storage.objects.create",
    "storage.objects.delete",
    "storage.objects.get",
    "storage.objects.list"
  ]
}

resource "google_project_iam_member" "storage-sa-role-binding" {
  project = "<gcp_project_id>"
  role = google_project_iam_custom_role.storage-custom-role.id
  member = "serviceAccount:${google_service_account.storage-sa.email}"
  condition {
    title       = "bucket_conditions"
    description = "Bucket conditions."
    expression  = "resource.name.startsWith(\"projects/_/buckets/${google_storage_bucket.storage-bucket.name}/\") || resource.name == \"projects/_/buckets/${google_storage_bucket.storage-bucket.name}\""
  }
}

resource "google_project_iam_custom_role" "deployment-custom-role" {
  role_id     = "tftestdeploymentrole"
  title       = "CelerData Compute Engine Deployment Role"
  description = "Custom role with only necessary permissions for CelerData"
  project     = "<gcp_project_id>"

  permissions = [
    "iam.serviceAccounts.actAs",
    "storage.buckets.get"
  ]
}

resource "google_project_iam_member" "deployment-sa-custom-role-binding-1" {
  project = "<gcp_project_id>"
  role    = google_project_iam_custom_role.deployment-custom-role.id
  member  = "serviceAccount:service@celerdata-byoc-1683716900563.iam.gserviceaccount.com"
}

resource "google_project_iam_member" "deployment-sa-static-role-binding" {
  project = "<gcp_project_id>"
  role    = "roles/compute.admin"
  member  = "serviceAccount:service@celerdata-byoc-1683716900563.iam.gserviceaccount.com"
}

resource "google_compute_network" "celerdata_vpc" {
  name                    = "tf-test-vpc"
  auto_create_subnetworks = false
  routing_mode            = "REGIONAL"
  bgp_best_path_selection_mode = "LEGACY"
}

resource "google_compute_subnetwork" "subnetwork" {
  name          = "tf-test-subnetwork"
  ip_cidr_range = "10.0.0.0/24"
  region        = us-central1
  network       = google_compute_network.celerdata_vpc.name
  private_ip_google_access = true
}

resource "google_compute_firewall" "fr-ingress-internal" {
  name    = "tf-test-fr-ingress-internal"
  network = google_compute_network.celerdata_vpc.name
  priority = 100
  direction = "INGRESS"

  allow {
    protocol = "all"
  }
  target_tags = [local.gcp_network_tag]
  source_ranges = ["10.0.0.0/24"]
}

resource "google_compute_firewall" "fr-ingress-external" {
  name    = "tf-test-fr-ingress-external"
  network = google_compute_network.celerdata_vpc.name
  priority = 100
  direction = "INGRESS"

  allow {
    protocol = "tcp"
    ports = ["443", "9030"]
  }
  source_ranges = ["0.0.0.0/0"]
}

resource "google_compute_firewall" "fr-egress-internal" {
  name    = "tf-test-fr-egress-internal"
  network = google_compute_network.celerdata_vpc.name
  priority = 100
  direction = "EGRESS"

  allow {
    protocol = "all"
  }
  target_tags = [local.gcp_network_tag]
  source_ranges = ["10.0.0.0/24"]
}

resource "google_compute_firewall" "fr-egress-external" {
  name    = "tf-test-fr-egress-external"
  network = google_compute_network.celerdata_vpc.name
  priority = 100
  direction = "EGRESS"

  allow {
    protocol = "tcp"
    ports = ["443","9030"]
  }
  source_ranges = ["0.0.0.0/0"]
}

resource "google_compute_firewall" "fr-ingress-nlb" {
  name    = "tf-test-fr-ingress-nlb"
  network = google_compute_network.celerdata_vpc.name
  priority = 100
  direction = "INGRESS"

  allow {
    protocol = "tcp"
    ports = ["443"]
  }
  target_tags = [local.gcp_network_tag]
  source_ranges = ["35.191.0.0/16", "209.85.152.0/22", "209.85.204.0/22", "130.211.0.0/22"]
}

See the following documents for more information:

• google_project_service
• google_storage_bucket
• google_service_account
• google_project_iam_custom_role
• google_project_iam_member
• google_compute_network
• google_compute_firewall

Describe infrastructure

This section provides a sample infrastructure configuration that automates the deployment of an elastic (shared-data) CelerData cluster on GCP to help you understand how you can work with the CelerData Cloud BYOC provider. It assumes that you have completed the preparations, configured the providers, and configured the GCP objects.

To create an elastic CelerData cluster, you need to declare the following resources, which represent the infrastructure to be built, in the .tf file (for example, main.tf) in which you have configured the providers and GCP objects.

celerdatabyoc_gcp_data_credential

resource "celerdatabyoc_gcp_data_credential" "storage_credential" {
  name = "tf-test-0604"
  bucket_name = google_storage_bucket.storage-bucket.name
  service_account = google_service_account.storage-sa.email
}

This resource contains the following required arguments:

• name: (Forces new resource) The name of the data credential. Enter a unique name.

• bucket_name: (Forces new resource) The name of the GCP bucket. Set this argument to google_storage_bucket.storage-bucket.name.

• service_account: (Forces new resource) The service account email of the storage. Set this argument to google_service_account.storage-sa.email.

celerdatabyoc_gcp_deployment_credential

resource "celerdatabyoc_gcp_deployment_credential" "deployment_credential" {
  name = "tf-test-0604"
  project_id = "<gcp_project_id>"
}

This resource contains the following required arguments:

• name: (Forces new resource) The name of the deployment credential. Enter a unique name.

• project_id: (Forces new resource) The service account email of the storage. Set this argument to google_service_account.storage-sa.email.

celerdatabyoc_gcp_network

resource "celerdatabyoc_gcp_network" "network_credential" {
  name = "tf-test-0604"
  region = us-central1
  subnet = google_compute_subnetwork.subnetwork.name
  network_tag = "<gcp_network_tag>"
  deployment_credential_id = celerdatabyoc_gcp_deployment_credential.deployment_credential.id
}

This resource contains the following required arguments and optional arguments:

Required:

• name: (Forces new resource) The name of the network configuration. Enter a unique name.

• region: (Forces new resource) The ID of the GCP region. See Supported cloud platforms and regions.

• subnet: (Forces new resource) The GCP subnet. e.g., google_compute_subnetwork.subnetwork.name.

• network_tag: (Forces new resource) The target tag of the firewall rules that you use to enable connectivity between cluster nodes within your own VPC, and between CelerData's VPC and your own VPC over TLS.

• deployment_credential_id: (Forces new resource) The ID of the deployment credential. Set this argument to celerdatabyoc_gcp_deployment_credential.example.id.

Optional:

• psc_connection_id: (Forces new resource) The ID of the Private Service Connection that you create to allow direct, secure connectivity between CelerData's VPC and your own VPC. For information about how to create a PSC Connection, see Create a Private Service Connect Endpoint.

  NOTE: If you do not specify a PSC Connection, CelerData's VPC communicates with your own VPC over the Internet.

Setup CelerData cluster-related resources

resource "celerdatabyoc_elastic_cluster_v2" "elastic_cluster" {
  deployment_credential_id = celerdatabyoc_gcp_deployment_credential.deployment_credential.id
  data_credential_id       = celerdatabyoc_gcp_data_credential.storage_credential.id
  network_id               = celerdatabyoc_gcp_network.network_credential.id

  cluster_name = "<cluster_name>"
  coordinator_node_size = "<coordinator_node_instance_type>"
  coordinator_node_count = <coordinator_node_number>

  default_warehouse {
    compute_node_size        = "<compute_node_instance_type>"
    compute_node_count       = <compute_node_number>

    // optional
    compute_node_volume_config {
      vol_number = <vol_number>
      vol_size = <vol_size>
      iops = <iops>
      throughput = <throughput>
    }
    // optional
    compute_node_configs = {
        <key> = <value>
    }
  }

  default_admin_password = "<SQL_user_initial_password>"
  expected_cluster_state = "{Suspended | Running}"
  resource_tags = {
    celerdata = "<tag_name>"
  }
  idle_suspend_interval = 60
  csp    = "gcp"
  region = "us-central1"
}

Apply configurations

After you finish configuring the providers and describing the infrastructure objects in your Terraform configuration, follow these steps to apply the configuration in your Terraform project:

1. Initialize and install the providers defined in the Terraform configuration:

   terraform init

2. Verify that your Terraform project has been properly configured:

   terraform plan

   If there are any errors, edit the Terraform configuration and re-run the preceding command.

3. Apply the Terraform configuration:

   terraform apply

When the system returns an "Apply complete!" message, the Terraform configuration has been successfully applied.

note

After you change the provider versions in the Terraform configuration, you must run terraform init -upgrade to initialize the providers and then run terraform apply again to apply the Terraform configuration.

Delete configurations

You can delete your Terraform configuration if you no longer need it.

Deleting a Terraform configuration means destroying all resources created by the CelerData Cloud BYOC provider.

To delete a Terraform configuration, run the following command in your Terraform project:

terraform destroy

When the system returns a "Destroy complete!" message, the Terraform configuration has been successfully deleted and the cluster created by the Terraform configuration is also released.