Connect to Google Cloud Secret Manager

This article describes how to connect an instance of Google Cloud Secret Manager as a secret store to your Humanitec Operator setup.

Before you begin

Before you begin, make sure you have the following resources and permissions:

You have the Secret Manager API enabled in a Google Cloud project.
You have determined the required access level (read/write/no access) to the secret store for the Platform Orchestrator and the Humanitec Operator, respectively. See Architecture for guidance.
You have installed and configured the Humanitec Operator on a Google Kubernetes Engine (GKE) cluster.
(Recommended) Workload Identity Federation is enabled on your GKE cluster and on the node pool running the Humanitec Operator.
You have access to the cluster via kubectl.
You have the gcloud CLI installed and have authenticated against Google Cloud.
You have reviewed the Implications of switching to Operator mode and prepared accordingly. Only relevant if this is the first secret store to be connected to your Organization.

Prepare your environment

Define the Google Cloud Project ID for which you wish to use the Secret Manager API. This may be the same project that is housing your GKE cluster, or a different one.

export GSM_PROJECT_ID=<my-gsm-project-id>

Define the Google Cloud Project ID that is housing your GKE cluster. This may be the same project for which you wish to use the Secret Manager API, or a different one.

export GKE_PROJECT_ID=<my-gke-project-id>

Define an ID for your secret store. This ID will later match the secret store registrations in the Operator and in the Platform Orchestrator.

export SECRET_STORE_ID=my-gsm

Enable Workload Identity for the Humanitec Operator

Using Workload Identity (this section) is the recommended option to authenticate the Humanitec Operator. It frees you from the need to manage any service account access credentials. If you do not wish to use Workload Identity, skip to the next section.

Workload Identity Federation allows workloads in your GKE clusters to access Google Cloud services, including the Secret Manager API, in a secure and manageable way.

Workload Identity Federation lets you bind a role directly to a Kubernetes service account without the need to provide an IAM service account first.

Define the Google Cloud Project number of the project housing your GKE cluster:

export GKE_PROJECT_NUMBER=$(gcloud projects describe ${GKE_PROJECT_ID} --format='get(projectNumber)')

Prepare the member name for the IAM policy binding:

export IAM_POLICY_BINDING_MEMBER=principal://iam.googleapis.com/projects/${GKE_PROJECT_NUMBER}/locations/global/workloadIdentityPools/${GKE_PROJECT_ID}.svc.id.goog/subject/ns/humanitec-operator-system/sa/humanitec-operator-controller-manager

Proceed to add a policy binding for the service account.

Prepare service account credentials for the Humanitec Operator

Using Workload Identity instead of a self-managed service account is the recommended option to authenticate the Humanitec Operator. It frees you from the need to manage any service account access credentials. To use Workload Identity, preform the steps in the previous section and skip this one.

To grant the Operator access to the Secret Manager API, create a Google service account (GSA), create credentials for this service account, and then place those credentials in a Kubernetes Secret. This Secret will then be referenced in the secret store definition.

Prepare environment variables for the service account:

export HUMANITEC_OPERATOR_GSA_NAME=humanitec-operator-gsa
export HUMANITEC_OPERATOR_GSA_ID=${HUMANITEC_OPERATOR_GSA_NAME}@${GKE_PROJECT_ID}.iam.gserviceaccount.com

Create an IAM service account:

gcloud iam service-accounts create ${HUMANITEC_OPERATOR_GSA_NAME} \
    	--display-name=${HUMANITEC_OPERATOR_GSA_NAME}

Create access keys for the service account:

gcloud iam service-accounts keys create ${HUMANITEC_OPERATOR_GSA_NAME}.json \
    	--iam-account ${HUMANITEC_OPERATOR_GSA_ID}

Create a Kubernetes Secret containing the access keys:

kubectl create secret generic my-gcpsm-secret \
	-n humanitec-operator-system \
	--from-file=secret-access-credentials=${HUMANITEC_OPERATOR_GSA_NAME}.json

Prepare the member name for the IAM policy binding:

export IAM_POLICY_BINDING_MEMBER=serviceAccount:${HUMANITEC_OPERATOR_GSA_ID}

Add policy binding

Add the proper policy binding to allow the Operator service account access to the Secret Manager API.

For read access (the Secret Manager is not your default secret store), add this policy binding:

gcloud projects add-iam-policy-binding "${GSM_PROJECT_ID}" \
    --member "${IAM_POLICY_BINDING_MEMBER}" \
    --role 'roles/secretmanager.secretAccessor'

For read/write access (the Secret Manager is your default secret store), add this policy binding:

Option 1: Using a built-in role:

gcloud projects add-iam-policy-binding "${GSM_PROJECT_ID}" \
    --member "${IAM_POLICY_BINDING_MEMBER}" \
    --role 'roles/secretmanager.admin'

Option 2: Creating and using a least privilege custom role with the required permissions only:

gcloud iam roles create secretmanager.readwrite \
--project "${GSM_PROJECT_ID}" \
--title "Secret Reader/Writer" \
--description "Can create new and update existing secrets and read them" \
--permissions 'secretmanager.secrets.create,secretmanager.secrets.delete,secretmanager.secrets.update,secretmanager.versions.add,secretmanager.versions.access,secretmanager.versions.list'

gcloud projects add-iam-policy-binding "${GSM_PROJECT_ID}" \
    --member "${IAM_POLICY_BINDING_MEMBER}" \
    --role "projects/${GSM_PROJECT_ID}/roles/secretmanager.readwrite"

Register the secret store with the Operator

Create the secret store registration using the following command. Modify it according to your setup:

If this is not your default secret store, omit the label app.humanitec.io/default-store (What is the default secret store?).
Choose the variant that matches the selected authentication method for the Humanitec Operator.

kubectl apply -f - << EOF
apiVersion: humanitec.io/v1alpha1
kind: SecretStore
metadata:
  name: ${SECRET_STORE_ID}
  namespace: humanitec-operator-system
  labels:
    app.humanitec.io/default-store: "true"
spec:
  gcpsm:
    projectID: ${GSM_PROJECT_ID}
    auth: {}
EOF

kubectl apply -f - << EOF
apiVersion: humanitec.io/v1alpha1
kind: SecretStore
metadata:
  name: ${SECRET_STORE_ID}
  namespace: humanitec-operator-system
  labels:
    app.humanitec.io/default-store: "true"
spec:
  gcpsm:
    projectID: ${GSM_PROJECT_ID}
    auth:
      secretAccessKeySecretRef:
        name: my-gcpsm-secret
        key: secret-access-credentials
EOF

Confirm the secret store registration.

To confirm the secret store registration, and anytime you wish to check for registered secret stores, use the following command:

kubectl get secretstores -n humanitec-operator-system

Register the secret store with the Platform Orchestrator

The Platform Orchestrator automatically switches to “Humanitec Operator” mode when configured with an external secret store marked primary. See the Architecture guidance for the details and implications of that switch.

The Platform Orchestrator needs to know which secret store to reference in the custom resources it creates for the Operator. This step is therefore always required. To begin using your secret store, you must now register it with the Platform Orchestrator.

The ID of the secret store in the Platform Orchestrator must match the ID of the SecretStore object created in the cluster.

(Optional) Prepare write access for the Platform Orchestrator.

Skip this step if you do not wish to grant write access to your Secret Manager via the Platform Orchestrator.

Create an IAM service account:

gcloud iam service-accounts create writeonly-secrets --project ${GSM_PROJECT_ID}

Create an IAM role with write-only permissions to Secret Manager:

gcloud iam roles create secretmanager.writer \
--project "${GSM_PROJECT_ID}" \
--title "Secret Writer" \
--description "Can create new and update existing secrets but not read them" \
--permissions 'secretmanager.secrets.create,secretmanager.secrets.delete,secretmanager.secrets.update,secretmanager.versions.add'

Create an IAM policy binding:

gcloud projects add-iam-policy-binding "${GSM_PROJECT_ID}" \
--member "serviceAccount:writeonly-secrets@${GSM_PROJECT_ID}.iam.gserviceaccount.com" \
--role "projects/${GSM_PROJECT_ID}/roles/secretmanager.writer"

Create an access key for the IAM service account:

gcloud iam service-accounts keys create writeonly-secrets-sa-key.json \
  --iam-account "writeonly-secrets@${GSM_PROJECT_ID}.iam.gserviceaccount.com"

Set the ID of your Humanitec Organization (must be all lowercase):

export HUMANITEC_ORG=my-humanitec-org-id

Set a Humanitec API token:

export HUMANITEC_TOKEN=my-humanitec-api-token

Run this command to register the secret store. Modify it according to your setup:

Omit the "auth" part if you are not granting write access via the Platform Orchestrator.
Set "primary" to false if this is not your primary secret store (What is the primary secret store?).

humctl api post /orgs/${HUMANITEC_ORG}/secretstores \
  -d '{
  "id": "'"${SECRET_STORE_ID}"'",
  "primary": true,
  "gcpsm": {
    "project_id": "'"${GSM_PROJECT_ID}"'",
    "auth": {
      "secret_access_key": '"$(cat writeonly-secrets-sa-key.json | jq "@json")"'
    }
  }
}'

curl https://api.humanitec.io/orgs/${HUMANITEC_ORG}/secretstores \
  -X POST \
  -H "Authorization: Bearer $HUMANITEC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "id": "'"${SECRET_STORE_ID}"'",
  "primary": true,
  "gcpsm": {
    "project_id": "'"${GSM_PROJECT_ID}"'",
    "auth": {
      "secret_access_key": '"$(cat writeonly-secrets-sa-key.json | jq "@json")"'
    }
  }
}'

resource "humanitec_secretstore" "gcpsm" {
  id      = "my-gsm"
  primary = true
  gcpsm = {
    project_id = var.gcp_project_id
    auth = {
      # Needs to be a single line string e.g. created via `cat writeonly-secrets-sa-key.json | jq "@json"`
      secret_access_key = var.gcp_secretmanager_secret_access_key
    }
  }
}

If, at a later stage, you need to update the secret for an already registered secret store, the following command can be used.

humctl api patch /orgs/${HUMANITEC_ORG}/secretstores/${SECRET_STORE_ID} \
  -d '{
  "gcpsm": {
    "project_id": "'"${GSM_PROJECT_ID}"'",
    "auth": {
      "secret_access_key": '"$(cat writeonly-secrets-sa-key.json | jq "@json")"'
    }
  }
}'

curl https://api.humanitec.io/orgs/${HUMANITEC_ORG}/secretstores/${SECRET_STORE_ID} \
  -X PATCH \
  -H "Authorization: Bearer $HUMANITEC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "gcpsm": {
    "project_id": "'"${GSM_PROJECT_ID}"'",
    "auth": {
      "secret_access_key": '"$(cat writeonly-secrets-sa-key.json | jq "@json")"'
    }
  }
}'

Perform another terraform apply using the definition shown above, and current variable values.

Make sure all referenced environment variables are set and that the writeonly-secrets-sa-key.json file exists containing an updated key, according to the previous steps.

To disable access for the Platform Orchestrator, use the same command and set "secret_access_key": "{}" in the "auth" section.

Confirm the secret store registration.

To confirm the secret store registration, and anytime you wish to check for registered secret stores, use the following command:

humctl api get /orgs/${HUMANITEC_ORG}/secretstores

curl -s https://api.humanitec.io/orgs/${HUMANITEC_ORG}/secretstores \
  -H "Authorization: Bearer $HUMANITEC_TOKEN" \
  | jq

Resource cookies

Resource cookies are stored in the Secret Manager according to this pattern:

projects/<project-id>/secrets/resources--active--<resource-id>--cookies--<driver>

Troubleshooting

ACCESS_TOKEN_SCOPE_INSUFFICIENT

If you receive an error message via the Platform Orchestrator reading ACCESS_TOKEN_SCOPE_INSUFFICIENT when deploying an Application, Workload Identity might not be set up correctly for your GKE cluster, or for the node pool where the Operator is running. Check the Google Cloud Workload Identity documentation and perform the verification step described therein.

Next steps

Test the Humanitec Operator installation using these test cases.
Perform the Update Resource Definitions for related Applications tutorial to verify your setup.
- Ensure the sample Applications are being deployed to your Orchestrator-enabled cluster by adjusting the matching criteria for the cluster Resource.
- Observe how custom resources of type workload and resource are being created on the cluster, and check their status sections.
- Observe how the Operator writes operational state (“resource cookies”) into your default secret store.

Connect to Google Cloud Secret Manager

Before you begin #

Prepare your environment #

Enable Workload Identity for the Humanitec Operator #

Prepare service account credentials for the Humanitec Operator #

Add policy binding #

Register the secret store with the Operator #

Register the secret store with the Platform Orchestrator #

Resource cookies #

Troubleshooting #

ACCESS_TOKEN_SCOPE_INSUFFICIENT #

Next steps #