Connect to Google Cloud Secret Manager

This article describes how to connect an instance of Google Cloud Secret Manager as a secret store to your Humanitec Operator setup.

Before you begin

Before you begin, make sure you have the following resources and permissions:

You have the Secret Manager API enabled in a Google Cloud project.
You have determined the required access level (read/write/no access) to the secret store for the Platform Orchestrator and the Humanitec Operator, respectively. See Architecture for guidance.
You have installed and configured the Humanitec Operator on a Google Kubernetes Engine (GKE) cluster.
Workload Identity is enabled on your GKE cluster and on the node pool running the Humanitec Operator.
You have access to the cluster via kubectl.
You have the gcloud CLI installed and have authenticated against Google Cloud.

Prepare your environment

Define the Google Cloud Project ID for which you wish to use the Secret Manager API. This may be the same project that is housing your GKE cluster, or a different one.

export GSM_PROJECT_ID=<my-gsm-project-id>

Define the Google Cloud Project ID that is housing your GKE cluster. This may be the same project for which you wish to use the Secret Manager API, or a different one.

export GKE_PROJECT_ID=<my-gke-project-id>

Define an ID for your secret store. This ID will later match the secret store registrations in the Operator and in the Platform Orchestrator.

export SECRET_STORE_ID=my-gsm

Enable Workload Identity for the Humanitec Operator

Workload Identity allows workloads in your GKE clusters to impersonate Identity and Access Management (IAM) service accounts to access Google Cloud services.

To grant the Operator access to the Secret Manager API, create a Google Service Account (GSA) and then enable Workload Identity for the Humanitec Operator.

Prepare environment variables for the Service Account:

export HUMANITEC_OPERATOR_GSA_NAME=humanitec-operator-gsa
export HUMANITEC_OPERATOR_GSA_ID=${HUMANITEC_OPERATOR_GSA_NAME}@${GKE_PROJECT_ID}.iam.gserviceaccount.com

Create an IAM service account and policy binding:

gcloud iam service-accounts create ${HUMANITEC_OPERATOR_GSA_NAME}

Create a policy binding for this Service Account:

gcloud iam service-accounts add-iam-policy-binding "${HUMANITEC_OPERATOR_GSA_ID}" \
  --member "serviceAccount:${GKE_PROJECT_ID}.svc.id.goog[humanitec-operator-system/humanitec-operator-controller-manager]" \
  --role 'roles/iam.workloadIdentityUser'

Upgrade the Helm installation of the Humanitec Operator to set the Annotation iam.gke.io/gcp-service-account to the Kubernetes Service Account it uses:

helm upgrade \
  humanitec-operator \
  oci://registry.humanitec.io/charts/humanitec-operator \
  -n humanitec-operator-system \
  --set "controllerManager.serviceAccount.annotations.iam\.gke\.io/gcp-service-account=${HUMANITEC_OPERATOR_GSA_ID}"

(Add the --version <version> parameter to pin the Helm chart to a particular version.)

If you are using an IaC tool to provision the Helm chart, make sure to add the new Helm value to the configuration.

Add the proper policy binding to allow the Operator Service Account access to the Secret Manager API.

For read access, add this policy binding:

gcloud projects add-iam-policy-binding "${GSM_PROJECT_ID}" \
    --member "serviceAccount:${HUMANITEC_OPERATOR_GSA_ID}" \
    --role 'roles/secretmanager.secretAccessor'

For read/write access, add this policy binding:

Option 1: Using a built-in role:

gcloud projects add-iam-policy-binding "${GSM_PROJECT_ID}" \
    --member "serviceAccount:${HUMANITEC_OPERATOR_GSA_ID}" \
    --role 'roles/secretmanager.admin'

Option 2: Creating and using a least privilege custom role with the required permissions only:

gcloud iam roles create secretmanager.readwrite \
--project "${GSM_PROJECT_ID}" \
--title "Secret Reader/Writer" \
--description "Can create new and update existing secrets and read them" \
--permissions 'secretmanager.secrets.create,secretmanager.secrets.delete,secretmanager.secrets.update,secretmanager.versions.add,secretmanager.versions.access'

gcloud projects add-iam-policy-binding "${GSM_PROJECT_ID}" \
    --member "serviceAccount:${HUMANITEC_OPERATOR_GSA_ID}" \
    --role "projects/${GSM_PROJECT_ID}/roles/secretmanager.readwrite"

Register the secret store with the Operator

Create the secret store registration using the following command. Modify it according to your setup:

If this is not your default secret store, omit the label app.humanitec.io/default-store (What is the default secret store?).

kubectl apply -f - << EOF
apiVersion: humanitec.io/v1alpha1
kind: SecretStore
metadata:
  name: ${SECRET_STORE_ID}
  namespace: humanitec-operator-system
  labels:
    app.humanitec.io/default-store: "true"
spec:
  gcpsm:
    projectID: ${GSM_PROJECT_ID}
    auth: {}
EOF

Confirm the secret store registration.

To confirm the secret store registration, and anytime you wish to check for registered secret stores, use the following command:

kubectl get secretstores -n humanitec-operator-system

Register the secret store with the Platform Orchestrator

The Platform Orchestrator automatically switches to “Humanitec Operator” mode when configured with an external secret store marked primary. See the Architecture guidance for the details and implications of that switch.

The Platform Orchestrator needs to know which secret store to reference in the custom resources it creates for the Operator. This step is therefore always required. To begin using your secret store, you must now register it with the Platform Orchestrator.

The ID of the secret store in the Platform Orchestrator must match the ID of the SecretStore object created in the cluster.

(Optional) Prepare write access for the Platform Orchestrator.

Skip this step if you do not wish to grant write access to your Secret Manager via the Platform Orchestrator.

Create an IAM service account:

gcloud iam service-accounts create writeonly-secrets --project ${GSM_PROJECT_ID}

Create an IAM role with write-only permissions to Secret Manager:

gcloud iam roles create secretmanager.writer \
--project "${GSM_PROJECT_ID}" \
--title "Secret Writer" \
--description "Can create new and update existing secrets but not read them" \
--permissions 'secretmanager.secrets.create,secretmanager.secrets.delete,secretmanager.secrets.update,secretmanager.versions.add'

Create an IAM policy binding:

gcloud projects add-iam-policy-binding "${GSM_PROJECT_ID}" \
--member "serviceAccount:writeonly-secrets@${GSM_PROJECT_ID}.iam.gserviceaccount.com" \
--role "projects/${GSM_PROJECT_ID}/roles/secretmanager.writer"

Create an access key for the IAM service account:

gcloud iam service-accounts keys create writeonly-secrets-sa-key.json \
  --iam-account "writeonly-secrets@${GSM_PROJECT_ID}.iam.gserviceaccount.com"

Set the ID of your Humanitec Organization (must be all lowercase):

export HUMANITEC_ORG=my-humanitec-org-id

Set a Humanitec API token:

export HUMANITEC_TOKEN=my-humanitec-api-token

Run this command to register the secret store. Modify it according to your setup:

Omit the "auth" part if you are not granting write access via the Platform Orchestrator.
Set "primary" to false if this is not your primary secret store (What is the primary secret store?).

curl https://api.humanitec.io/orgs/${HUMANITEC_ORG}/secretstores \
  -X POST \
  -H "Authorization: Bearer $HUMANITEC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "id": "'"${SECRET_STORE_ID}"'",
  "primary": true,
  "gcpsm": {
    "project_id": "'"${GSM_PROJECT_ID}"'",
    "auth": {
      "secret_access_key": '"$(cat writeonly-secrets-sa-key.json | jq "@json")"'
    }
  }
}'

If, at a later stage, you need to update the secret for an already registered secret store, the following command can be used.

curl https://api.humanitec.io/orgs/${HUMANITEC_ORG}/secretstores/${SECRET_STORE_ID} \
  -X PATCH \
  -H "Authorization: Bearer $HUMANITEC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "gcpsm": {
    "project_id": "'"${GSM_PROJECT_ID}"'",
    "auth": {
      "secret_access_key": '"$(cat writeonly-secrets-sa-key.json | jq "@json")"'
    }
  }
}'

Make sure all referenced environment variables are set and that the writeonly-secrets-sa-key.json file exists containing an updated key, according to the previous steps.

To disable access for the Platform Orchestrator, use the same command and set "secret_access_key": "{}" in the "auth" section.

Confirm the secret store registration.

To confirm the secret store registration, and anytime you wish to check for registered secret stores, use the following command:

curl -s https://api.humanitec.io/orgs/${HUMANITEC_ORG}/secretstores \
  -H "Authorization: Bearer $HUMANITEC_TOKEN" \
  | jq

Resource cookies

Resource cookies are stored in the Secret Manager according to this pattern:

projects/<project-id>/secrets/resources--active--<resource-id>--cookies--<driver>

Troubleshooting

ACCESS_TOKEN_SCOPE_INSUFFICIENT

If you receive an error message via the Platform Orchestrator reading ACCESS_TOKEN_SCOPE_INSUFFICIENT when deploying an Application, Workload Identity might not be set up correctly for your GKE cluster, or for the node pool where the Operator is running. Check the Google Cloud Workload Identity documentation and perform the verification step described therein.

Next steps

Perform the Update Resource Definitions for related Applications tutorial to verify your setup.

Ensure the sample Applications are being deployed to your Orchestrator-enabled cluster by adjusting the matching criteria for the cluster Resource.
Observe how custom resources of type workload and resource are being created on the cluster, and check their status sections.
Observe how the Operator writes operational state (“resource cookies”) into your default secret store.

Connect to Google Cloud Secret Manager

Before you begin #

Prepare your environment #

Enable Workload Identity for the Humanitec Operator #

Register the secret store with the Operator #

Register the secret store with the Platform Orchestrator #

Resource cookies #

Troubleshooting #

ACCESS_TOKEN_SCOPE_INSUFFICIENT #

Next steps #