Gcp Pubsub

Example: gcp-pubsub resource based on Google Cloud Pub/Sub #

Configuration #

This example configures a gcp-pubsub-topic and a gcp-pubsub-subscription Resource Definition using Google Cloud Pub/Sub.

Those Resource Definitions can be used in your Score file using:

# publishing workload
containers:
  app:
    ...
    variables:
      TOPIC_NAME: ${resources.topic.name}
resources:
  ...
  topic:
    type: gcp-pubsub-topic
    class: basic-publisher
    # Make it a shared resource by using the "id"
    id: main-topic

# subscribing workload
containers:
  app:
    ...
    variables:
      SUBSCRIPTION_NAME: ${resources.subscription.name}
resources:
  ...
  subscription:
    type: gcp-pubsub-subscription
    class: basic-subscriber
    params:
      topic_name: ${resources['gcp-pubsub-topic.basic#shared.main-topic'].outputs.name}

Infrastructure setup #

The workload service account will be automatically assigned to the necessary roles with the selected policies.

graph TD;
  topic["GCP Pub/Sub topic"]
  sub["GCP Pub/Sub subscription"]
  subgraph GKE Cluster
    topic_pod[workload pod]
    topic_service[k8s service account]
    sub_pod[workload pod]
    sub_service[k8s service account]
  end
  topic_service -- bind role on --> topic
  topic_service --> topic_pod
  topic --> topic_pod
  sub_service -- bind role on --> sub
  sub_service --> sub_pod
  sub --> sub_pod
  sub --> topic

Orchestrator setup #

The Resource Graph is using delegator resources to expose shared resources with different access policies.

graph LR;
  workload_1 --> delegator_1["id: shared.topic_1, resource_type: gcp-pubsub-topic, class: basic-publisher"] --> shared.gcp-pubsub-topic_1["id: shared.topic_1, resource_type: gcp-pubsub-topic, class: basic"]
  workload_2 --> delegator_2["id: externals.workload_2, resource_type: gcp-pubsub-subscription, class: basic-subscriber"] --> shared.gcp-pubsub-subscriber_1["id: externals.workload_2, resource_type: gcp-pubsub-subscription, class: basic"]
  workload_2 --> shared.delegator_1["id: shared.subscription_1, resource_type: gcp-pubsub-subscription, class: basic-subscriber"]
  workload_3 --> shared.delegator_1 --> shared.gcp-pubsub-subscriber_2["id: shared.subscription_1, resource_type: gcp-pubsub-subscription, class: basic"]

In this example, the Workload workload_1 acts as a publisher. It requests a shared resource of type gcp-pubsub-topic and class basic-publisher. This will be a “delegator” resource, co-provisioning an access policy (not shown) and referencing a “base” resource to provision the actual topic.

The Workload workload_2 is a subscriber. It requests a private resource of type gcp-pubsub-subscription. This will likewise be a “delegator” resource, co-provisioning an access policy (not shown) and referencing a “base” resource to provision the actual subscription.

Both workload_2 and workload_3 request another shared resource of type gcp-pubsub-subscription. This will again be a “delegator” resource. The two Workloads will effectively share the “base” subscription resource, and since they are using the same delegator resource.

Terraform docs #

Requirements #

Providers #

Modules #

Resources #

Inputs #

main.tf (view on GitHub) :

# GCP service account used by Humanitec to provision resources

resource "google_service_account" "humanitec_provisioner" {
  account_id  = var.name
  description = "Account used by Humanitec to provision resources"
}

resource "google_project_iam_member" "humanitec_provisioner" {
  project = var.project
  role    = "roles/owner"
  member  = "serviceAccount:${google_service_account.humanitec_provisioner.email}"
}

resource "google_service_account_key" "humanitec_provisioner" {
  service_account_id = google_service_account.humanitec_provisioner.name
}

resource "humanitec_resource_account" "humanitec_provisioner" {
  id   = var.name
  name = var.name
  type = "gcp"

  credentials = base64decode(google_service_account_key.humanitec_provisioner.private_key)

  depends_on = [
    # Otherwise the account looses permissions before the resources are deleted
    google_project_iam_member.humanitec_provisioner
  ]
}

# Example application and resource definition criteria

resource "humanitec_application" "example" {
  id   = var.name
  name = var.name
}

# GCP pubsub

locals {
  # Classes used to build the resource definition graph
  gpt_basic_class             = "basic"
  gps_basic_class             = "basic"
  gpt_publisher_policy_class  = "gcp-pubsub-subscription-basic-publisher"
  gps_subscriber_policy_class = "gcp-pubsub-subscription-basic-subscriber"

  # Classes that developers can select from
  gpt_basic_publisher_class  = "basic-publisher"
  gps_basic_subscriber_class = "basic-subscriber"
}

# Required resources for workload identity

module "k8s_service_account_workload" {
  source = "github.com/humanitec-architecture/resource-packs-gcp?ref=v2024-06-14//humanitec-resource-defs/k8s-service-account/workload"

  resource_packs_gcp_url = var.resource_packs_gcp_url
  resource_packs_gcp_rev = var.resource_packs_gcp_rev
  append_logs_to_error   = true
  driver_account         = humanitec_resource_account.humanitec_provisioner.id

  project = var.project
  prefix  = var.prefix
}

resource "humanitec_resource_definition_criteria" "k8s_service_account_workload" {
  resource_definition_id = module.k8s_service_account_workload.id
  app_id                 = humanitec_application.example.id
  force_delete           = true
}

module "workload" {
  source = "github.com/humanitec-architecture/resource-packs-gcp?ref=v2024-06-14//humanitec-resource-defs/workload/service-account"

  prefix = var.prefix
}

resource "humanitec_resource_definition_criteria" "workload" {
  resource_definition_id = module.workload.id
  app_id                 = humanitec_application.example.id
  force_delete           = true
}

providers.tf (view on GitHub) :

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.17"
    }
    humanitec = {
      source  = "humanitec/humanitec"
      version = "~> 1.0"
    }
  }

  required_version = ">= 1.3.0"
}

provider "humanitec" {}

provider "google" {
  project = var.project

  default_labels = {
    "managed_by" = "terraform"
    "source"     = "github.com/humanitec-architecture/resource-pack-gcp"
  }
}

pubsub_subscription.tf (view on GitHub) :

# gcp pubsub subscription related resources

module "pubsub_subscription_basic" {
  source = "github.com/humanitec-architecture/resource-packs-gcp?ref=v2024-06-14//humanitec-resource-defs/gcp-pubsub-subscription/basic"

  resource_packs_gcp_url = var.resource_packs_gcp_url
  resource_packs_gcp_rev = var.resource_packs_gcp_rev
  append_logs_to_error   = true
  driver_account         = humanitec_resource_account.humanitec_provisioner.id

  project = var.project
  prefix  = var.prefix
}

resource "humanitec_resource_definition_criteria" "pubsub_subscription_basic" {
  resource_definition_id = module.pubsub_subscription_basic.id
  app_id                 = humanitec_application.example.id
  class                  = local.gps_basic_class
  force_delete           = true
}

# policy

module "iam_role_binding_gcp_pubsub_subscription_subscriber" {
  source = "github.com/humanitec-architecture/resource-packs-gcp?ref=v2024-06-14//humanitec-resource-defs/gcp-iam-policy-binding/basic"

  prefix = var.prefix
  name   = "gcp-pubsub-subscription-basic-subscriber"

  type        = "pubsub_subscription"
  scope_key   = "subscription"
  scope_value = "$${resources['gcp-pubsub-subscription.${local.gps_basic_class}'].outputs.name}"
  role        = "roles/pubsub.subscriber"
}

resource "humanitec_resource_definition_criteria" "iam_role_binding_gcp_pubsub_subscription_subscriber" {
  resource_definition_id = module.iam_role_binding_gcp_pubsub_subscription_subscriber.id
  app_id                 = humanitec_application.example.id
  class                  = local.gps_subscriber_policy_class
  force_delete           = true
}

## Exposed delegator resource definition

module "gps_basic_subscriber" {
  source = "github.com/humanitec-architecture/resource-packs-gcp?ref=v2024-06-14//humanitec-resource-defs/gcp-pubsub-subscription/delegator"

  prefix = var.prefix

  gps_resource_class    = local.gps_basic_class
  policy_resource_class = local.gps_subscriber_policy_class
}

resource "humanitec_resource_definition_criteria" "gps_basic_subscriber" {
  resource_definition_id = module.gps_basic_subscriber.id
  app_id                 = humanitec_application.example.id
  class                  = local.gps_basic_subscriber_class
  force_delete           = true
}

pubsub_topic.tf (view on GitHub) :

# gcp pubsub topic related resources

module "pubsub_topic_basic" {
  source = "github.com/humanitec-architecture/resource-packs-gcp?ref=v2024-06-14//humanitec-resource-defs/gcp-pubsub-topic/basic"

  resource_packs_gcp_url = var.resource_packs_gcp_url
  resource_packs_gcp_rev = var.resource_packs_gcp_rev
  append_logs_to_error   = true
  driver_account         = humanitec_resource_account.humanitec_provisioner.id

  project = var.project
  prefix  = var.prefix
}

resource "humanitec_resource_definition_criteria" "pubsub_topic_basic" {
  resource_definition_id = module.pubsub_topic_basic.id
  app_id                 = humanitec_application.example.id
  class                  = local.gpt_basic_class
  force_delete           = true
}

# policy

module "iam_role_binding_gcp_pubsub_topic_publisher" {
  source = "github.com/humanitec-architecture/resource-packs-gcp?ref=v2024-06-14//humanitec-resource-defs/gcp-iam-policy-binding/basic"

  prefix = var.prefix
  name   = "gcp-pubsub-topic-basic-publisher"

  type        = "pubsub_topic"
  scope_key   = "topic"
  scope_value = "$${resources['gcp-pubsub-topic.${local.gpt_basic_class}'].outputs.name}"
  role        = "roles/pubsub.publisher"
}

resource "humanitec_resource_definition_criteria" "iam_role_binding_gcp_pubsub_topic_publisher" {
  resource_definition_id = module.iam_role_binding_gcp_pubsub_topic_publisher.id
  app_id                 = humanitec_application.example.id
  class                  = local.gpt_publisher_policy_class
  force_delete           = true
}

## Exposed delegator resource definition

module "gpt_basic_publisher" {
  source = "github.com/humanitec-architecture/resource-packs-gcp?ref=v2024-06-14//humanitec-resource-defs/gcp-pubsub-topic/delegator"

  prefix = var.prefix

  gpt_resource_class    = local.gpt_basic_class
  policy_resource_class = local.gpt_publisher_policy_class
}

resource "humanitec_resource_definition_criteria" "gpt_basic_publisher" {
  resource_definition_id = module.gpt_basic_publisher.id
  app_id                 = humanitec_application.example.id
  class                  = local.gpt_basic_publisher_class
  force_delete           = true
}

terraform.tfvars.example (view on GitHub) :

# Name of the example application
name = "hum-rp-gcp-pubsub-example"

prefix = "hum-rp-gcp-pubsub-ex-"

# GCP project ID
project = ""

# GCP Resource Pack git ref
resource_packs_gcp_rev = "refs/tags/v2024-06-14"

# GCP Resource Pack git url
resource_packs_gcp_url = "https://github.com/humanitec-architecture/resource-packs-gcp.git"

variables.tf (view on GitHub) :

variable "project" {
  description = "GCP project ID"
  type        = string
}

variable "resource_packs_gcp_url" {
  description = "GCP Resource Pack git url"
  type        = string
  default     = "https://github.com/humanitec-architecture/resource-packs-gcp.git"
}

variable "resource_packs_gcp_rev" {
  description = "GCP Resource Pack git ref"
  type        = string
  default     = "refs/tags/v2024-06-14"
}

variable "name" {
  description = "Name of the example application"
  type        = string
  default     = "hum-rp-gcp-pubsub-example"
}

variable "prefix" {
  type    = string
  default = "hum-rp-gcp-pubsub-ex-"
}