Skip to main content

Provision CelerData Cloud BYOC on GCP

Step by step deployment of a CelerData Cloud BYOC cluster on GCP:

Read this article before you start a Terraform configuration for your cluster deployment on GCP.

Preparations​

Before using the CelerData Cloud BYOC provider to create infrastructure at the GCP account level for the first time, you must complete the following preparations:

GCP prerequisites​

  1. Have an GCP service account with administrative privileges.
  2. Install the GCP CLI. For more information, see How to install the GCP CLI.

CelerData prerequisites​

Obtain the credentials to authenticate into the CelerData Cloud BYOC platform. For details, see Authentication.

Terraform prerequisites​

  1. Install Terraform in your terminal.

  2. Have a Terraform project. In your terminal, create an empty directory (for example, terraform) and then switch to it. (Each separate set of Terraform configuration files must be in its own directory, which is called a Terraform project.)

Configure providers​

This section assumes that you have completed the preparations.

Create a .tf file (for example, main.tf) in your Terraform project. Then, add the following code snippet to the .tf file:

terraform {
  required_providers {
    celerdatabyoc = {
      source  = "CelerData/celerdatabyoc"
      version = "<provider_version>"
    }
    google = {
      source = "hashicorp/google"
      version = "6.37.0"
    }
  }
}

provider "celerdatabyoc" {
  client_id = "<client_id>"
  client_secret = "<client_secret>"
}

provider "google" {
  project     = "<project_id>"
  region      = "<region>"
}

The parameters you need to specify are as follows:

  • provider_version: Enter the CelerData provider version of your choice. We recommend that you select the latest provider version, for example version = "1.0.3". You can view the provider versions offered by CelerData Cloud BYOC from the CelerData Cloud BYOC provider page.
  • client_id and client_secret: Enter the Client ID and Secret of your application key. See For CelerData.
  • project_id: Enter your GCP cloud project ID.
  • region: Enter the ID of the GCP cloud region in which you want your CelerData cluster to run. See Supported cloud platforms and regions.

Configure GCP objects​

The deployment on GCP requires creating a Data Credential, Deployment Credential, and Network Credential in CelerData Cloud. For detailed steps, please refer to: Cloud Setting For GCP

Here is an example written based on the document:

provider "google" {
  project     = "<gcp_project_id>"
  region      = "us-central1"
}

resource "google_project_service" "project" {
  for_each = toset([
    "analyticshub.googleapis.com",
    "bigquery.googleapis.com",
    "bigqueryconnection.googleapis.com",
    "bigquerydatapolicy.googleapis.com",
    "bigquerymigration.googleapis.com",
    "bigqueryreservation.googleapis.com",
    "bigquerystorage.googleapis.com",
    "compute.googleapis.com",
    "cloudapis.googleapis.com",
    "logging.googleapis.com",
    "iamcredentials.googleapis.com",
    "dataplex.googleapis.com",
    "datastore.googleapis.com",
    "dns.googleapis.com",
    "monitoring.googleapis.com",
    "oslogin.googleapis.com",
    "sqladmin.googleapis.com",
    "storage.googleapis.com",
    "cloudtrace.googleapis.com",
    "dataform.googleapis.com",
    "storage-api.googleapis.com",
    "privilegedaccessmanager.googleapis.com",
    "servicedirectory.googleapis.com",
    "servicemanagement.googleapis.com",
    "servicenetworking.googleapis.com",
    "serviceusage.googleapis.com",
  ])
  service            = each.key
  disable_on_destroy = false
}

resource "google_storage_bucket" "storage-bucket" {
  name     = "tf-test-bucket-${local.gcp_project_id}"
  location = us-central1
  storage_class = "STANDARD"
  uniform_bucket_level_access = false
  public_access_prevention    = "enforced"

  force_destroy = false

  labels = {
    environment = "prod"
  }

  lifecycle {
    ignore_changes = [
      cors
    ]
  }
}

resource "google_service_account" "storage-sa" {
  account_id   = "tf-test-storage-sa"
  display_name = "Terraform Test Service Account"
}

resource "google_project_iam_custom_role" "storage-custom-role" {
  role_id     = "tfteststoragerole"
  title       = "CelerData Compute Engine Storage Role"
  description = "Custom role with only necessary permissions for CelerData"
  project     = "<gcp_project_id>"

  permissions = [
    "iam.serviceAccounts.signBlob",
    "storage.buckets.get",
    "storage.buckets.update",
    "storage.objects.create",
    "storage.objects.delete",
    "storage.objects.get",
    "storage.objects.list"
  ]
}

resource "google_project_iam_member" "storage-sa-role-binding" {
  project = "<gcp_project_id>"
  role = google_project_iam_custom_role.storage-custom-role.id
  member = "serviceAccount:${google_service_account.storage-sa.email}"
  condition {
    title       = "bucket_conditions"
    description = "Bucket conditions."
    expression  = "resource.name.startsWith(\"projects/_/buckets/${google_storage_bucket.storage-bucket.name}/\") || resource.name == \"projects/_/buckets/${google_storage_bucket.storage-bucket.name}\""
  }
}

resource "google_project_iam_custom_role" "deployment-custom-role" {
  role_id     = "tftestdeploymentrole"
  title       = "CelerData Compute Engine Deployment Role"
  description = "Custom role with only necessary permissions for CelerData"
  project     = "<gcp_project_id>"

  permissions = [
    "iam.serviceAccounts.actAs",
    "storage.buckets.get"
  ]
}

resource "google_project_iam_member" "deployment-sa-custom-role-binding-1" {
  project = "<gcp_project_id>"
  role    = google_project_iam_custom_role.deployment-custom-role.id
  member  = "serviceAccount:service@celerdata-byoc-1683716900563.iam.gserviceaccount.com"
}

resource "google_project_iam_member" "deployment-sa-static-role-binding" {
  project = "<gcp_project_id>"
  role    = "roles/compute.admin"
  member  = "serviceAccount:service@celerdata-byoc-1683716900563.iam.gserviceaccount.com"
}

resource "google_compute_network" "celerdata_vpc" {
  name                    = "tf-test-vpc"
  auto_create_subnetworks = false
  routing_mode            = "REGIONAL"
  bgp_best_path_selection_mode = "LEGACY"
}

resource "google_compute_subnetwork" "subnetwork" {
  name          = "tf-test-subnetwork"
  ip_cidr_range = "10.0.0.0/24"
  region        = us-central1
  network       = google_compute_network.celerdata_vpc.name
  private_ip_google_access = true
}

resource "google_compute_firewall" "fr-ingress-internal" {
  name    = "tf-test-fr-ingress-internal"
  network = google_compute_network.celerdata_vpc.name
  priority = 100
  direction = "INGRESS"

  allow {
    protocol = "all"
  }
  target_tags = [local.gcp_network_tag]
  source_ranges = ["10.0.0.0/24"]
}

resource "google_compute_firewall" "fr-ingress-external" {
  name    = "tf-test-fr-ingress-external"
  network = google_compute_network.celerdata_vpc.name
  priority = 100
  direction = "INGRESS"

  allow {
    protocol = "tcp"
    ports = ["443", "9030"]
  }
  source_ranges = ["0.0.0.0/0"]
}

resource "google_compute_firewall" "fr-egress-internal" {
  name    = "tf-test-fr-egress-internal"
  network = google_compute_network.celerdata_vpc.name
  priority = 100
  direction = "EGRESS"

  allow {
    protocol = "all"
  }
  target_tags = [local.gcp_network_tag]
  source_ranges = ["10.0.0.0/24"]
}

resource "google_compute_firewall" "fr-egress-external" {
  name    = "tf-test-fr-egress-external"
  network = google_compute_network.celerdata_vpc.name
  priority = 100
  direction = "EGRESS"

  allow {
    protocol = "tcp"
    ports = ["443","9030"]
  }
  source_ranges = ["0.0.0.0/0"]
}

resource "google_compute_firewall" "fr-ingress-nlb" {
  name    = "tf-test-fr-ingress-nlb"
  network = google_compute_network.celerdata_vpc.name
  priority = 100
  direction = "INGRESS"

  allow {
    protocol = "tcp"
    ports = ["443"]
  }
  target_tags = [local.gcp_network_tag]
  source_ranges = ["35.191.0.0/16", "209.85.152.0/22", "209.85.204.0/22", "130.211.0.0/22"]
}

See the following documents for more information:

Describe infrastructure​

This section provides a sample infrastructure configuration that automates the deployment of a classic (shared-nothing) CelerData cluster on GCP to help you understand how you can work with the CelerData Cloud BYOC provider. It assumes that you have completed the preparations, configured the providers, and configured the GCP objects.

To create a classic CelerData cluster, you need to declare the following resources, which represent the infrastructure to be built, in the .tf file (for example, main.tf) in which you have configured the providers and GCP objects.

celerdatabyoc_gcp_data_credential​

resource "celerdatabyoc_gcp_data_credential" "storage_credential" {
  name = "tf-test-0604"
  bucket_name = google_storage_bucket.storage-bucket.name
  service_account = google_service_account.storage-sa.email
}

This resource contains the following required arguments:

  • name: (Forces new resource) The name of the data credential. Enter a unique name.

  • bucket_name: (Forces new resource) The name of the GCP bucket. Set this argument to google_storage_bucket.storage-bucket.name.

  • service_account: (Forces new resource) The service account email of the storage. Set this argument to google_service_account.storage-sa.email.

celerdatabyoc_gcp_deployment_credential​

resource "celerdatabyoc_gcp_deployment_credential" "deployment_credential" {
  name = "tf-test-0604"
  project_id = "<gcp_project_id>"
}

This resource contains the following required arguments:

  • name: (Forces new resource) The name of the deployment credential. Enter a unique name.

  • project_id: (Forces new resource) The service account email of the storage. Set this argument to google_service_account.storage-sa.email.

celerdatabyoc_gcp_network​

resource "celerdatabyoc_gcp_network" "network_credential" {
  name = "tf-test-0604"
  region = us-central1
  subnet = google_compute_subnetwork.subnetwork.name
  network_tag = "<gcp_network_tag>"
  deployment_credential_id = celerdatabyoc_gcp_deployment_credential.deployment_credential.id
}

This resource contains the following required arguments and optional arguments:

Required:

  • name: (Forces new resource) The name of the network configuration. Enter a unique name.

  • region: (Forces new resource) The ID of the GCP region. See Supported cloud platforms and regions.

  • subnet: (Forces new resource) The GCP subnet. e.g., google_compute_subnetwork.subnetwork.name.

  • network_tag: (Forces new resource) The target tag of the firewall rules that you use to enable connectivity between cluster nodes within your own VPC, and between CelerData's VPC and your own VPC over TLS.

  • deployment_credential_id: (Forces new resource) The ID of the deployment credential. Set this argument to celerdatabyoc_gcp_deployment_credential.example.id.

Optional:

  • psc_connection_id: (Forces new resource) The ID of the Private Service Connection that you create to allow direct, secure connectivity between CelerData's VPC and your own VPC. For information about how to create a PSC Connection, see Create a Private Service Connect Endpoint.

    NOTE: If you do not specify a PSC Connection, CelerData's VPC communicates with your own VPC over the Internet.

celerdatabyoc_classic_cluster​

resource "celerdatabyoc_classic_cluster" "demo_cluster" {
  deployment_credential_id = celerdatabyoc_gcp_deployment_credential.deployment_credential.id
  data_credential_id       = celerdatabyoc_gcp_data_credential.storage_credential.id
  network_id               = celerdatabyoc_gcp_network.network_credential.id

  cluster_name = "<cluster_name>"
  fe_instance_type = "<fe_node_instance_type>"
  fe_node_count = 1

  be_instance_type = "<be_node_instance_type>"
  be_node_count = 1
  // optional
  be_volume_config {
    vol_number = <vol_number>
    vol_size = <vol_size>
    iops = <iops>
    throughput = <throughput>
  }

  default_admin_password = "<SQL_user_initial_password>"
  expected_cluster_state = "Running"
  resource_tags = {
    celerdata = "<tag_name>"
  }
  csp    = "gcp"
  region = "us-central1"
}

The celerdatabyoc_classic_cluster resource contains the following required arguments and optional arguments:

Required:

  • cluster_name: (Forces new resource) The desired name for the cluster.

  • fe_instance_type: The instance type for FE nodes in the cluster. Select an FE instance type from the table "Supported instance types."

  • deployment_credential_id: (Forces new resource) The ID of the deployment credential. Set the value to celerdatabyoc_gcp_deployment_credential.deployment_credential.id.

  • data_credential_id: (Forces new resource) The ID of the data credential. Set the value to celerdatabyoc_gcp_data_credential.storage_credential.id.

  • network_id: (Forces new resource) The ID of the network configuration. Set the value to celerdatabyoc_gcp_network.network_credential.id.

  • be_instance_type: The instance type for BE nodes in the cluster. Select a BE instance type from the table "Supported instance types."

  • default_admin_password: The initial password of the cluster admin user.

  • expected_cluster_state: When creating a cluster, you need to declare the status of the cluster you are creating. Cluster states are categorized as Suspended and Running. If you want the cluster to start after provisioning, set this argument to Running. If you do not do so, the cluster will be suspended after provisioning.

  • csp: The cloud service provider of the cluster. Set this argument to gcp.

  • region: The ID of the GCP region to which the VPC hosting the cluster belongs. See Supported cloud platforms and regions.

Optional:

  • fe_node_count: The number of FE nodes in the cluster. Valid values: 1, 3, and 5. Default value: 1.

  • be_node_count: The number of BE nodes in the cluster. Valid values: any non-zero positive integer. Default value: 3.

  • be_disk_number: (Forces new resource) The maximum number of disks that are allowed for each BE. Valid values: [1,24]. Default value: 2.

  • be_disk_per_size: The size per disk for each BE. Unit: GB. Maximum value: 16000. Default value: 100. You can only increase the value of this parameter, and the time interval between two value changes must be greater than 6 hours.

  • resource_tags: The tags to be attached to the cluster.

Apply configurations​

After you finish configuring the providers and describing the infrastructure objects in your Terraform configuration, follow these steps to apply the configuration in your Terraform project:

  1. Initialize and install the providers defined in the Terraform configuration:

    terraform init

  2. Verify that your Terraform project has been properly configured:

    terraform plan

    If there are any errors, edit the Terraform configuration and re-run the preceding command.

  3. Apply the Terraform configuration:

    terraform apply

When the system returns an "Apply complete!" message, the Terraform configuration has been successfully applied.

note

After you change the provider versions in the Terraform configuration, you must run terraform init -upgrade to initialize the providers and then run terraform apply again to apply the Terraform configuration.

Delete configurations​

You can delete your Terraform configuration if you no longer need it.

Deleting a Terraform configuration means destroying all resources created by the CelerData Cloud BYOC provider.

To delete a Terraform configuration, run the following command in your Terraform project:

terraform destroy

When the system returns a "Destroy complete!" message, the Terraform configuration has been successfully deleted and the cluster created by the Terraform configuration is also released.