Connect to AWS Secrets Manager

This article describes how to connect an instance of AWS Secrets Manager as a secret store to your Humanitec Operator setup.

Before you begin

Before you begin, make sure you have the following resources and permissions:

You have determined the required access level (read/write/no access) to the secret store for the Platform Orchestrator and the Humanitec Operator, respectively. See Architecture for guidance.
You have installed and configured the Humanitec Operator on an Amazon Elastic Kubernetes Service (EKS) cluster.
(Recommended) An IAM Role assignment mechanism for Kubernetes service accounts is enabled for your EKS cluster, which can be either EKS Pod Identities or IAM Roles for Service Accounts (IRSA).
You have access to the cluster via kubectl.
You have the aws CLI installed and have authenticated against AWS.
You have reviewed the Implications of switching to Operator mode and prepared accordingly. Only relevant if this is the first secret store to be connected to your Organization.

The instructions assume that the identity being used for the Operator (IAM User or IAM Role) is in the same account as the Secrets Manager. We currently do not support Cross-account access.

Prepare your environment

Define the AWS Account ID where the Secrets Manager is located:

export SECRETS_MANAGER_ACCOUNT_ID=<account-id>

Define the AWS Region where the Secrets Manager is located. This region can be different from the one your EKS cluster is running in:

export SECRETS_MANAGER_REGION=<region>

Define an ID for your secret store. This ID will later match the secret store registrations in the Operator and in the Platform Orchestrator.

export SECRET_STORE_ID=my-awssm

Define the name of your EKS cluster:

export EKS_NAME=<my-eks-cluster-name>

Prepare a policy for the Humanitec Operator

Prepare an identity-based permissions policy that defines the required access to the Secrets Manager for the Operator. This policy will subsqeuently be attached to the IAM identity the Operator will be using.

Adjust the policy if required:

The policy, as shown, facilitates read/write access for the Operator. If you wish to grant only read access because the target Secrets Manager is not your default secret store, remove all elements in the Action except "secretsmanager:GetSecretValue".
If you are organizing your secrets hierarchically inside Secrets Manager and want to grant access to a specific path only, adjust the Resource value to specify the path, e.g. "Resource":...㊙️somePath/*". See Permissions policy examples for AWS Secrets Manager for more details.

cat <<EOF > policy-operator.json
{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",
        "secretsmanager:CreateSecret",
        "secretsmanager:DeleteSecret",
        "secretsmanager:PutSecretValue"
      ],
    "Resource": "arn:aws:secretsmanager:${SECRETS_MANAGER_REGION}:${SECRETS_MANAGER_ACCOUNT_ID}㊙️*"
  }
}
EOF

Create the policy:

export OPERATOR_POLICY_NAME=humanitec-operator-${SECRETS_MANAGER_REGION}-${EKS_NAME}

export OPERATOR_POLICY_ARN=$(aws iam create-policy \
  --policy-name ${OPERATOR_POLICY_NAME} \
  --policy-document file://policy-operator.json \
  --region ${SECRETS_MANAGER_REGION} | jq -r .Policy.Arn )
echo $OPERATOR_POLICY_ARN

Use an IAM User for the Humanitec Operator

Using an IAM User will require you to create and maintain static credentials for that user. An IAM Role assignment mechanism for Kubernetes service accounts (EKS Pod Identities or IRSA) instead of an IAM User is the recommended option to authenticate the Humanitec Operator. It frees you from the need to manage any access credentials. If you do wish to use an IAM User, proceed here, otherwise skip to the next section.

To grant the Operator access to the Secrets Manager, create an IAM User, create access keys for this user, and then place those keys in a Kubernetes Secret. This Secret will then be referenced in the secret store definition.

Prepare environment variables for the user:

export OPERATOR_IAM_USER_NAME=humanitec-operator-${SECRETS_MANAGER_REGION}-${EKS_NAME}

Create a new IAM User:

aws iam create-user \
  --user-name ${OPERATOR_IAM_USER_NAME} \
  --region ${SECRETS_MANAGER_REGION}

Create an access key:

export OPERATOR_USER_ACCESS_KEY=$(aws iam create-access-key --user-name ${OPERATOR_IAM_USER_NAME})

Create a Kubernetes Secret in the Humanitec Operator namespace containing the access key data:

export SECRET_NAME=awssm-credentials

kubectl create secret generic ${SECRET_NAME} \
  -n humanitec-operator-system \
  --from-literal=access_key_id=$(echo $OPERATOR_USER_ACCESS_KEY | jq -r .AccessKey.AccessKeyId) \
  --from-literal=secret_access_key=$(echo $OPERATOR_USER_ACCESS_KEY | jq -r .AccessKey.SecretAccessKey)

Attach the policy you created previously to the IAM User used by the Operator:

aws iam attach-user-policy \
  --user-name ${OPERATOR_IAM_USER_NAME} \
  --region ${SECRETS_MANAGER_REGION} \
  --policy-arn ${OPERATOR_POLICY_ARN}

Configure IAM Role and service account

Perform the steps in this section if you are using an IAM Role assignment mechanism for Kubernetes service accounts (EKS Pod Identities or IRSA). If you wish to use an IAM User with static credentials instead, head to the previous section.

Both EKS Pod Identities and IRSA let a Kubernetes workload effectively assume an IAM Role via the Kubernetes services account a Pod is running with. The instructions assume that you have prepared the required setup as per the product documentation for one of these methods and can now continue to configure the IAM Role and service account.

Prepare environment variables:

export EKS_REGION=<the-clusters-region>
export IAM_ROLE_NAME=humanitec-operator-${SECRETS_MANAGER_REGION}-${EKS_NAME}
export IAM_ROLE_DESCRIPTION="Humanitec Operator service account on EKS cluster ${EKS_NAME} in ${SECRETS_MANAGER_REGION}"

Set your cluster’s OIDC identity provider:

# Step is not required for EKS Pod Identities

export OIDC_PROVIDER=$(aws eks describe-cluster --name ${EKS_NAME} \
  --region ${EKS_REGION} \
  --query "cluster.identity.oidc.issuer" \
  --output text | sed -e "s/^https:\/\///")

Create a trust policy file for the IAM role:

cat >trust-relationship.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ]
        }
    ]
}
EOF

cat >trust-relationship.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::$SECRETS_MANAGER_ACCOUNT_ID:oidc-provider/$OIDC_PROVIDER"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "$OIDC_PROVIDER:aud": "sts.amazonaws.com",
          "$OIDC_PROVIDER:sub": "system:serviceaccount:humanitec-operator-system:humanitec-operator-controller-manager"
        }
      }
    }
  ]
}
EOF

Create the IAM role and capture its ARN:

export IAM_ROLE_ARN=$(aws iam create-role --role-name ${IAM_ROLE_NAME} \
  --assume-role-policy-document file://trust-relationship.json \
  --description "${IAM_ROLE_DESCRIPTION}" | jq -r .Role.Arn )

Attach the policy you created previously to the IAM role used by the Operator:

aws iam attach-role-policy \
  --role-name ${IAM_ROLE_NAME} \
  --policy-arn=${OPERATOR_POLICY_ARN}

Associate the IAM role with the service account:

Run the following command to create the association:

aws eks create-pod-identity-association \
  --cluster-name ${EKS_NAME} \
  --role-arn ${IAM_ROLE_ARN} \
  --namespace humanitec-operator-system \
  --service-account humanitec-operator-controller-manager

Upgrade the Helm installation of the Humanitec Operator to set the Annotation eks.amazonaws.com/role-arn to the IAM role it uses:

helm upgrade \
  humanitec-operator \
  oci://ghcr.io/humanitec/charts/humanitec-operator \
  -n humanitec-operator-system \
  --set "controllerManager.serviceAccount.annotations.eks\.amazonaws\.com/role-arn=${IAM_ROLE_ARN}"

(Add the --version <version> parameter to pin the Helm chart to a particular version.)

If you are using an IaC tool to provision the Helm chart, make sure to add the new Helm value to the configuration.

Restart the Operator Deployment to pick up the IAM role assignment:

kubectl rollout restart deployment \
  -n humanitec-operator-system \
  humanitec-operator-controller-manager

(Optional) Validate the IAM role assignment:

Perform the validation steps of the EKS Pod Identities documentation on the Humanitec Operator Pod.

Perform the validation steps of the IRSA documentation on the Humanitec Operator Pod.

Register the secret store with the Operator

Create the secret store registration using the following command. Modify it according to your setup:

If this is not your default secret store, omit the label app.humanitec.io/default-store (What is the default secret store?).

kubectl apply -f - << EOF
apiVersion: humanitec.io/v1alpha1
kind: SecretStore
metadata:
  name: ${SECRET_STORE_ID}
  namespace: humanitec-operator-system
  labels:
    app.humanitec.io/default-store: "true"
spec:
  awssm:
    region: ${SECRETS_MANAGER_REGION}
    auth:
      accessKeyIDSecretRef:
        name: ${SECRET_NAME}
        key: access_key_id
      secretAccessKeySecretRef:
        name: ${SECRET_NAME}
        key: secret_access_key
EOF

kubectl apply -f - << EOF
apiVersion: humanitec.io/v1alpha1
kind: SecretStore
metadata:
  name: ${SECRET_STORE_ID}
  namespace: humanitec-operator-system
  labels:
    app.humanitec.io/default-store: "true"
spec:
  awssm:
    region: ${SECRETS_MANAGER_REGION}
    auth: {}
EOF

Confirm the secret store registration.

To confirm the secret store registration, and anytime you wish to check for registered secret stores, use the following command:

kubectl get secretstores -n humanitec-operator-system

Register the secret store with the Platform Orchestrator

The Platform Orchestrator automatically switches to “Humanitec Operator” mode when configured with an external secret store marked primary. See the Architecture guidance for the details and implications of that switch.

The Platform Orchestrator needs to know which secret store to reference in the custom resources it creates for the Operator. This step is therefore always required. To begin using your secret store, you must now register it with the Platform Orchestrator.

The ID of the secret store in the Platform Orchestrator must match the ID of the SecretStore object created in the cluster.

(Optional) Prepare write access for the Platform Orchestrator.

Skip this step if you do not wish to grant write access to your Secrets Manager via the Platform Orchestrator.

Create an IAM User:

export ORCHESTRATOR_IAM_USER_NAME=platform-orchestrator-secretmanager-${SECRETS_MANAGER_REGION}

aws iam create-user \
  --user-name ${ORCHESTRATOR_IAM_USER_NAME} \
  --region ${SECRETS_MANAGER_REGION}

Create an access key:

export ORCHESTRATOR_USER_ACCESS_KEY=$(aws iam create-access-key --user-name ${ORCHESTRATOR_IAM_USER_NAME})

Prepare an identity-based permissions policy that defines the required access to the Secrets Manager for the Platform Orchestrator. This policy will be attached to the previously created IAM User. Note that it is a write-only policy.

Adjust the policy if required:

If you are organizing your secrets hierarchically inside Secrets Manager and want to grant access to a specific path only, adjust the Resource value to specify the path, e.g. "Resource":...㊙️somePath/*". See Permissions policy examples for AWS Secrets Manager for more details.

cat <<EOF > policy-orchestrator.json
{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
      "Action": [
        "secretsmanager:CreateSecret",
        "secretsmanager:DeleteSecret",
        "secretsmanager:PutSecretValue"
      ],
    "Resource": "arn:aws:secretsmanager:${SECRETS_MANAGER_REGION}:${SECRETS_MANAGER_ACCOUNT_ID}㊙️*"
  }
}
EOF

Create the policy:

export ORCHESTRATOR_POLICY_NAME=platform-orchestrator-${SECRETS_MANAGER_REGION}

export ORCHESTRATOR_POLICY_ARN=$(aws iam create-policy \
  --policy-name ${ORCHESTRATOR_POLICY_NAME} \
  --policy-document file://policy-orchestrator.json \
  --region ${SECRETS_MANAGER_REGION} | jq -r .Policy.Arn )
echo $ORCHESTRATOR_POLICY_ARN

Attach the policy to the IAM User used by the Platform Orchestrator:

aws iam attach-user-policy \
  --user-name ${ORCHESTRATOR_IAM_USER_NAME} \
  --region ${SECRETS_MANAGER_REGION} \
  --policy-arn ${ORCHESTRATOR_POLICY_ARN}

Set the ID of your Humanitec Organization (must be all lowercase):

export HUMANITEC_ORG=<my-humanitec-org-id>

Set a Humanitec API token:

export HUMANITEC_TOKEN=<my-humanitec-api-token>

Run this command to register the secret store. Modify it according to your setup:

Omit the "auth" part if you are not granting write access via the Platform Orchestrator.
Set "primary" to false if this is not your primary secret store (What is the primary secret store?).

humctl api post /orgs/${HUMANITEC_ORG}/secretstores \
  -d '{
  "id": "'"${SECRET_STORE_ID}"'",
  "primary": true,
  "awssm": {
    "region": "'"${SECRETS_MANAGER_REGION}"'",
    "auth": {
      "access_key_id": "'"$(echo $ORCHESTRATOR_USER_ACCESS_KEY | jq -r .AccessKey.AccessKeyId)"'",
      "secret_access_key": "'"$(echo $ORCHESTRATOR_USER_ACCESS_KEY | jq -r .AccessKey.SecretAccessKey)"'"
    }
  }
}'

curl https://api.humanitec.io/orgs/${HUMANITEC_ORG}/secretstores \
  -X POST \
  -H "Authorization: Bearer $HUMANITEC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "id": "'"${SECRET_STORE_ID}"'",
  "primary": true,
  "awssm": {
    "region": "'"${SECRETS_MANAGER_REGION}"'",
    "auth": {
      "access_key_id": "'"$(echo $ORCHESTRATOR_USER_ACCESS_KEY | jq -r .AccessKey.AccessKeyId)"'",
      "secret_access_key": "'"$(echo $ORCHESTRATOR_USER_ACCESS_KEY | jq -r .AccessKey.SecretAccessKey)"'"
    }
  }
}'

resource "humanitec_secretstore" "my_awssm" {
  id      = var.secret_store_id
  primary = true
  awssm = {
    region = var.secrets_manager_region
    auth = {
      access_key_id     = var.access_key_id
      secret_access_key = var.secret_access_key
    }
  }
}

If, at a later stage, you need to update the credentials for an already registered secret store, the following command can be used.

humctl api patch /orgs/${HUMANITEC_ORG}/secretstores/${SECRET_STORE_ID} \
  -d '{
  "awssm": {
    "region": "'"${SECRETS_MANAGER_REGION}"'",
    "auth": {
      "access_key_id": "'"$(echo $ORCHESTRATOR_USER_ACCESS_KEY | jq -r .AccessKey.AccessKeyId)"'",
      "secret_access_key": "'"$(echo $ORCHESTRATOR_USER_ACCESS_KEY | jq -r .AccessKey.SecretAccessKey)"'"
    }
  }
}'

curl https://api.humanitec.io/orgs/${HUMANITEC_ORG}/secretstores/${SECRET_STORE_ID} \
  -X PATCH \
  -H "Authorization: Bearer $HUMANITEC_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
  "awssm": {
    "region": "'"${SECRETS_MANAGER_REGION}"'",
    "auth": {
      "access_key_id": "'"$(echo $ORCHESTRATOR_USER_ACCESS_KEY | jq -r .AccessKey.AccessKeyId)"'",
      "secret_access_key": "'"$(echo $ORCHESTRATOR_USER_ACCESS_KEY | jq -r .AccessKey.SecretAccessKey)"'"
    }
  }
}'

Perform another terraform apply using the definition shown above, and current variable values.

Make sure all referenced environment variables are set according to the previous steps.

To disable access for the Platform Orchestrator, use the same command and set "secret_access_key": "xxx" in the "auth" section.

Confirm the secret store registration.

To confirm the secret store registration, and anytime you wish to check for registered secret stores, use the following command:

humctl api get /orgs/${HUMANITEC_ORG}/secretstores

curl -s https://api.humanitec.io/orgs/${HUMANITEC_ORG}/secretstores \
  -H "Authorization: Bearer $HUMANITEC_TOKEN" \
  | jq

Resource Cookies

Resource cookies are stored in the Secrets Manager according to this pattern:

resources/active/<resource-id>/cookies/<driver>

Next steps

Test the Humanitec Operator installation using these test cases.
Perform the Update Resource Definitions for related Applications tutorial to verify your setup.
- Ensure the sample Applications are being deployed to your Orchestrator-enabled cluster by adjusting the matching criteria for the cluster Resource.
- Observe how custom resources of type workload and resource are being created on the cluster, and check their status sections.
- Observe how the Operator writes operational state (“resource cookies”) into your default secret store.

Connect to AWS Secrets Manager

Before you begin #

Prepare your environment #

Prepare a policy for the Humanitec Operator #

Use an IAM User for the Humanitec Operator #

Configure IAM Role and service account #

Register the secret store with the Operator #

Register the secret store with the Platform Orchestrator #

Resource Cookies #

Next steps #