Connect to HashiCorp Vault

This article describes how to connect an instance of HashiCorp Vault as a secret store to your Humanitec Operator setup.

Before you begin

Before you begin, make sure you have the following resources and permissions:

An instance of HashiCorp Vault set up to use as your secret store.
The required access level (read/write/no access) to the secret store for the Platform Orchestrator and the Humanitec Operator, respectively. See Architecture for guidance.
The Humanitec Operator installed and configured on a Kubernetes (K8s) cluster.
Access to the cluster via kubectl.
The vault CLI installed (recommnded).
- All examples will use the vault CLI. You may choose to use another way of interacting with your Vault instance, e.g. via the UI or API, at your discretion.
You have reviewed the Implications of switching to Operator mode and prepared accordingly. Only relevant if this is the first secret store to be connected to your Organization.

Prepare your Vault

Prepare your local environment.

Set the address of your Vault for use by the CLI:

export VAULT_ADDR=https://<myvault.mydomain.example>

Define an ID for your secret store. This ID will later match the secret store registrations in the Humnanitec Operator and in the Platform Orchestrator.

export SECRET_STORE_ID=my-vault-store

Set the path where you’ll store your secrets in your Vault. We suggest the path secret for simplicity. If that path is already in use by another secrets engine in your Vault, choose a different path. Use “vault secrets list” to see all configured secrets engines.

export SECRETS_PATH=secret

vault login

Enable a secrets engine in your Vault with path secret and type kv.

vault secrets enable -version=2 -path=$SECRETS_PATH kv

The secrets engine must support versioning and thus be a version 2 engine to work. If you have an existing version 1 engine you wish to use, enable versioning on it by calling:

vault kv enable-versioning $SECRETS_PATH

Prepare the required policy for the Humanitec Operator on the selected path. See Architecture for guidance.

For read access (the Secret Manager is not your default secret store), prepare this policy:

export OPERATOR_ACCESS_LEVEL=read
cat << EOF > policy-$OPERATOR_ACCESS_LEVEL.hcl
path "$SECRETS_PATH/*" {
  capabilities = ["read", "list"]
}
EOF

For read/write access (the Secret Manager is your default secret store), prepare this policy:

export OPERATOR_ACCESS_LEVEL=read-write
cat << EOF > policy-$OPERATOR_ACCESS_LEVEL.hcl
path "$SECRETS_PATH/*" {
  capabilities = ["create", "update", "delete", "read", "list"]
}
EOF

Create the policy for the Humanitec Operator in your Vault:

vault policy write secret-$OPERATOR_ACCESS_LEVEL policy-$OPERATOR_ACCESS_LEVEL.hcl

Create the write access policy for the Platform Orchestrator on the selected path. Skip this step if the Orchestrator does not need access. See Architecture for guidance.

cat << EOF > policy-write.hcl
path "$SECRETS_PATH/*" {
  capabilities = ["create", "update", "delete"]
}
EOF

Create the policy for the Platform Orchestrator in your Vault:

vault policy write secret-write-only policy-write.hcl

Choose the authentication method for the Operator

The Humanitec Operator needs to authenticate against Vault and assume a specific role so that it can read and/or write secrets. Vault offers a wide range of authentication methods. Out of these, we describe the kubernetes and the token methods. Choose either one, and follow the instructions in the respective section.

All Vault tokens except root tokens have a TTL. Therefore if the token method is chosen, the secret must be periodically updated by an external process, or the token renewed. We therefore recommend using the kubernetes auth method approach which does not require any renewals.

Configure the kubernetes auth method

Skip this section if you’re using a different auth method.

The kubernetes auth method can be used to authenticate the Humanitec Operator against Vault using a K8s Service Account Token. Follow the steps below to configure your setup for this method.

Set the path where you’ll configure the auth method in your Vault. We currently only support the path kubernetes. To connect more than one cluster to the same Vault instance, choose another auth method for those clusters. Use “vault auth list” to see all currently configured auth methods and their paths.

export KUBERNETES_AUTH_PATH=kubernetes

Enable the authentication method kubernetes under the chosen path in your Vault.

vault auth enable -path=$KUBERNETES_AUTH_PATH kubernetes

Obtain your cluster’s public root CA certificate.

kubectl get cm kube-root-ca.crt -o jsonpath="{['data']['ca\.crt']}" > ca.crt

Set the variable KUBERNETES_API_SERVER to the API server endpoint of your cluster. If your kubeconfig’s current_context is set to your cluster, you can use:

export KUBERNETES_API_SERVER=$(kubectl config view --minify | yq .clusters[0].cluster.server) \
  && echo $KUBERNETES_API_SERVER

Write the K8s server data into the configuration of your Vault auth method.

vault write auth/$KUBERNETES_AUTH_PATH/config \
  kubernetes_host="$KUBERNETES_API_SERVER" \
  [email protected]

Create a role for your Operator in the Vault auth method.

vault write auth/$KUBERNETES_AUTH_PATH/role/humanitec-operator-role \
  bound_service_account_names=humanitec-operator-controller-manager \
  bound_service_account_namespaces=* \
  policies=secret-$OPERATOR_ACCESS_LEVEL \
  ttl=1h

Note that the role binds to the Service Account name humanitec-operator-controller-manager which is the Service Account of the Humanitec Operator. Having the role gets the Operator the access defined in the policy for the chosen access level.

Create the secret store registration. Modify it according to your setup:

If this is not your default secret store, omit the label app.humanitec.io/default-store (What is the default secret store?).

kubectl apply -f - << EOF
apiVersion: humanitec.io/v1alpha1
kind: SecretStore
metadata:
  name: ${SECRET_STORE_ID}
  namespace: humanitec-operator-system
  labels:
    app.humanitec.io/default-store: "true"
spec:
  vault:
    url: ${VAULT_ADDR}
    path: ${SECRETS_PATH}
    auth:
      role: humanitec-operator-role
EOF

Confirm the secret store registration.

To confirm the secret store registration, and anytime you wish to check for registered secret stores, use the following command:

kubectl get secretstores -n humanitec-operator-system

Configure the token auth method

Skip this section if you are using a different auth method.

The token auth method can be used to authenticate the Operator against Vault using a Vault token. This auth method is built-in and automatically available at the path /auth/token. Follow the steps below to configure your setup for this method.

A token with a limited TTL will have to be updated by an external process. Instructions to update the token are included below.

Create a Vault token.

Create a token attached to the proper policy you created earlier:

export VAULT_TOKEN_OPERATOR=$(vault token create -policy=secret-$OPERATOR_ACCESS_LEVEL -ttl=30d -field token)

The example uses a standard TTL approach with a 30-day validity. See the token create options for details on token expiration and TTL alternatives.

Place the token in a Kubernetes Secret.

Create a Kubernetes Secret containing the token:

kubectl create secret generic vault-token \
    -n humanitec-operator-system \
    --from-literal=token=${VAULT_TOKEN_OPERATOR}

If, at a later stage, you need to update the token for an already registered secret store (e.g. because it has reached its maximum TTL and needs to be renewed), place the new token value in the VAULT_TOKEN_OPERATOR environment variable and execute the following command:

kubectl patch secret vault-token \
  -n humanitec-operator-system \
  --patch="{\"data\": { \"token\": \"$(echo -n $VAULT_TOKEN_OPERATOR | base64 -w0)\"}}"

Create the secret store registration using the following command. Modify it according to your setup:

If this is not your default secret store, omit the label app.humanitec.io/default-store.

kubectl apply -f - << EOF
apiVersion: humanitec.io/v1alpha1
kind: SecretStore
metadata:
  name: ${SECRET_STORE_ID}
  namespace: humanitec-operator-system
  labels:
    app.humanitec.io/default-store: "true"
spec:
  vault:
    url: ${VAULT_ADDR}
    path: ${SECRETS_PATH}
    auth:
      tokenSecretRef:
        name: vault-token
        key: token
EOF

Confirm the secret store registration.

To confirm the secret store registration, and anytime you wish to check for registered secret stores, use the following command:

kubectl get secretstores -n humanitec-operator-system

Register the secret store with the Platform Orchestrator

The Platform Orchestrator automatically switches to “Humanitec Operator” mode when configured with an external secret store marked primary. See the Architecture guidance for the details and implications of that switch.

The Platform Orchestrator needs to know which secret store to reference in the custom resources it creates for the Operator. This step is therefore always required. To begin using your secret store, you must now register it with the Platform Orchestrator.

The ID of the secret store in the Platform Orchestrator must match the ID of the SecretStore object created in the cluster.

(Optional) Create a Vault token.

Skip this step if you do not wish to grant write access to your Vault via the Platform Orchestrator.

At this time, the Platform Orchestrator only supports the token authentication method for Vault.

A token with a limited TTL will have to be updated by an external process. Instructions to update the token are included below.

Create a token attached to the write-only policy you created earlier:

export VAULT_TOKEN_ORCHESTRATOR=$(vault token create -policy=secret-write-only -ttl=30d -field token)

The example uses a standard TTL approach with a 30-day validity. See the token create options for details on token expiration and TTL alternatives.

Set the ID of your Humanitec Organization (must be all lowercase):

export HUMANITEC_ORG=my-humanitec-org-id

Set a Humanitec API token:

export HUMANITEC_TOKEN=my-humanitec-api-token

Run this command to register the secret store. Modify it according to your setup:

Omit the "auth" part if you are not granting write access via the Platform Orchestrator.
Set "primary" to false if this is not your primary secret store (What is the primary secret store?).

humctl api post /orgs/${HUMANITEC_ORG}/secretstores \
  -d '{
  "id": "'${SECRET_STORE_ID}'",
  "primary": true,
  "vault": {
    "url": "'${VAULT_ADDR}'",
    "path": "'${SECRETS_PATH}'",
    "auth": {
      "token": "'${VAULT_TOKEN_ORCHESTRATOR}'"
    }
  }
}'

curl https://api.humanitec.io/orgs/${HUMANITEC_ORG}/secretstores \
  -X POST \
  -H "Authorization: Bearer $HUMANITEC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "id": "'${SECRET_STORE_ID}'",
  "primary": true,
  "vault": {
    "url": "'${VAULT_ADDR}'",
    "path": "'${SECRETS_PATH}'",
    "auth": {
      "token": "'${VAULT_TOKEN_ORCHESTRATOR}'"
    }
  }
}'

resource "humanitec_secretstore" "my-vault-store" {
  id      = "my-vault-store"
  primary = true
  vault = {
    url  = var.vault_url
    path = var.vault_secret_path
    auth = {
      token = var.vault_token
    }
  }
}

If, at a later stage, you need to update the token for an already registered secret store (e.g. because it has reached its maximum TTL and needs to be renewed), the following command can be used. Make sure all referenced environment variables are set according to the previous steps.

humctl api patch /orgs/${HUMANITEC_ORG}/secretstores/${SECRET_STORE_ID} \
  -d '{
  "vault": {
    "auth": {
      "token": "'${VAULT_TOKEN_ORCHESTRATOR}'"
    }
  }
}'

curl https://api.humanitec.io/orgs/${HUMANITEC_ORG}/secretstores/${SECRET_STORE_ID} \
  -X PATCH \
  -H "Authorization: Bearer $HUMANITEC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "vault": {
    "auth": {
      "token": "'${VAULT_TOKEN_ORCHESTRATOR}'"
    }
  }
}'

Perform another terraform apply using the definition shown above, and current variable values.

To disable access for the Platform Orchestrator, use the same command and set "token": "<some-random-value>" in the "auth" section.

Confirm the secret store registration.

To confirm the secret store registration, and anytime you wish to check for registered secret stores, use the following command:

humctl api get /orgs/${HUMANITEC_ORG}/secretstores

curl -s https://api.humanitec.io/orgs/${HUMANITEC_ORG}/secretstores \
  -H "Authorization: Bearer $HUMANITEC_TOKEN" \
  | jq

Resource cookies

Resource cookies are stored underneath the configured secret path according to this pattern:

<secret-path> / resources / active / <resource-id> / cookies

Next steps

Test the Humanitec Operator installation using these test cases.
Perform the Update Resource Definitions for related Applications tutorial to verify your setup.
- Ensure the sample Applications are being deployed to your Orchestrator-enabled cluster by adjusting the matching criteria for the cluster Resource.
- Observe how custom resources of type workload and resource are being created on the cluster, and check their status sections.
- Observe how the Operator writes operational state (“resource cookies”) into your default secret store.

Connect to HashiCorp Vault

Before you begin #

Prepare your Vault #

Choose the authentication method for the Operator #

Configure the kubernetes auth method #

Configure the token auth method #

Register the secret store with the Platform Orchestrator #

Resource cookies #

Next steps #

Before you begin

Prepare your Vault

Choose the authentication method for the Operator

Configure the kubernetes auth method

Configure the token auth method

Register the secret store with the Platform Orchestrator

Resource cookies

Next steps