Testing

Overview

This article contains test cases to verify the successful setup of the Humanitec Operator in your infrastructure. They are especially helpful after setting up the Humanitec Operator on a new Kubernetes cluster, or after connecting it to additional secret stores to ensure the connection to these stores works as expected.

We currently provide these tests:

Air-gapped: This test covers the toolchain “Humanitec Operator” - “secret store” only. You can use it when the Platform Orchestrator does not have access to your target Kubernetes cluster, e.g. when using a private cluster. Duration: 10-15 min.
End-to-end: This test covers the toolchain “Platform Orchestrator” - “Humanitec Operator” - “secret store”. You can use it when the Platform Orchestrator has access to your target Kubernetes cluster. It covers all of the objectives of the “Air-gapped” test, so you do not need to execute both. Duration: 20-25 min.

Prerequisites (all tests)

The Humanitec Operator is installed on the target cluster.
You have access to the target cluster via kubectl.
At least one secret store is configured in the Humanitec Operator, e.g. HashiCorp Vault. Select one target secret store to use in the tests if you have more than one.
You have write access to the target secret store to create demo secrets, e.g. via its respective CLI such as aws for AWS Secrets Manager, az for Azure Key Vault, gcloud for Google Cloud Secret Manager, or the vault CLI for HashiCorp Vault. The test cases will use the CLI tools, but you may choose to access your secret store using a different interface such as the UI.

Setup (all tests)

Perform these common setup steps before executing any of the tests.

Prepare your local environment. Replace all <placeholder values> with the values of your setup.

export SECRET_STORE_ID=<my-secret-store-id>
export APP_ID=my-app
export APP_NAME="My-App"

# Only when using AWS Secrets Manager
export SECRETS_MANAGER_REGION=<region> # E.g. eu-north-1

# Only when using Azure Key Vault
export KEY_VAULT_NAME=<my-keyvault-name>

# Only when using Google Secret Manager
export GSM_PROJECT_ID=<my-Google-Cloud-Secret-Manager-Project-ID>

# Only when using HashiCorp Vault
export SECRET_PATH=<my-secret-engine-path> # e.g. "secret"

Create secrets in your secret store. While not all secrets are used in all tests, we create all of them for simplification. We will delete them again when cleaning up.

aws secretsmanager create-secret \
  --name mysql-username --secret-string "super secret username" \
  --region ${SECRETS_MANAGER_REGION}
aws secretsmanager create-secret \
  --name mysql-password --secret-string "super secret password" \
  --region ${SECRETS_MANAGER_REGION}
aws secretsmanager create-secret \
  --name my-secret-shared-value --secret-string "super secret Shared Value" \
  --region ${SECRETS_MANAGER_REGION}

az keyvault secret set --name mysql-username \
  --vault-name ${KEY_VAULT_NAME} --value "super secret username"
az keyvault secret set --name mysql-password \
  --vault-name ${KEY_VAULT_NAME} --value "super secret password"
az keyvault secret set --name my-secret-shared-value \
  --vault-name ${KEY_VAULT_NAME} --value "super secret Shared Value"

echo -n "super secret username" | \
    gcloud secrets create mysql-username --project ${GSM_PROJECT_ID} --data-file=-
echo -n "super secret password" | \
    gcloud secrets create mysql-password --project ${GSM_PROJECT_ID} --data-file=-
echo -n "super secret Shared Value" | \
    gcloud secrets create my-secret-shared-value --project ${GSM_PROJECT_ID} --data-file=-

Expected output:

Created version [1] of the secret [mysql-username].
Created version [1] of the secret [mysql-password].
Created version [1] of the secret [my-secret-shared-value].

vault kv put ${SECRET_PATH}/mysql-username value="super secret username"
vault kv put ${SECRET_PATH}/mysql-password value="super secret password"
vault kv put ${SECRET_PATH}/my-secret-shared-value value="super secret Shared Value"

Expected output:

======== Secret Path ========
secret/data/mysql-username
======= Metadata =======
Key                Value
---                -----
created_time       2030-11-27T09:49:12.793035144Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

# (...likewise for mysql-password and my-secret-shared-value)

Test: Air-gapped

This test lets you create custom resources based on the Humanitec Operator CRDs straight on the target cluster, bypassing the need for the Platform Orchestrator. The Humanitec Operator will act on those custom resources and create the final Kubernetes manifests. No outbound network connectivity is required for the target cluster except accessing the secret store.

The custom resources contain secret references mapped to the secret store. The Humanitec Operator must resolve those secret references and read a secret value from the secret store as well as write a resource cookie to the default store.

Objectives

Verify the secret mapping and resource provisioning functionality of the Humanitec Operator on the target cluster.
Verify read access on the selected secret store for the Humanitec Operator.
Verify write access on the default secret store for the Humanitec Operator.

Prerequisites

All of the prerequisites for all tests.
You have performed the setup steps for all tests.

Test Steps

Create a namespace for the Application.

kubectl create namespace ${APP_ID}

Expected output:

namespace/my-app created

Create a SecretMapping custom resource.

cat << EOF | kubectl apply -n ${APP_ID} -f -
apiVersion: humanitec.io/v1alpha1
kind: SecretMapping
metadata:
  name: secretmapping-shared-secrets
spec:
  secretData:
  - remoteRef:
      key: my-secret-shared-value
      version: ""
    secretKey: my-secret-shared-value
    secretStoreName: ${SECRET_STORE_ID}
  secretName: shared-secrets
EOF

cat << EOF | kubectl apply -n ${APP_ID} -f -
apiVersion: humanitec.io/v1alpha1
kind: SecretMapping
metadata:
  name: secretmapping-shared-secrets
spec:
  secretData:
  - remoteRef:
      key: my-secret-shared-value
      version: ""
    secretKey: my-secret-shared-value
    secretStoreName: ${SECRET_STORE_ID}
  secretName: shared-secrets
EOF

cat << EOF | kubectl apply -n ${APP_ID} -f -
apiVersion: humanitec.io/v1alpha1
kind: SecretMapping
metadata:
  name: secretmapping-shared-secrets
spec:
  secretData:
  - remoteRef:
      key: my-secret-shared-value
      version: ""
    secretKey: my-secret-shared-value
    secretStoreName: ${SECRET_STORE_ID}
  secretName: shared-secrets
EOF

cat << EOF | kubectl apply -n ${APP_ID} -f -
apiVersion: humanitec.io/v1alpha1
kind: SecretMapping
metadata:
  name: secretmapping-shared-secrets
spec:
  secretData:
  - remoteRef:
      key: my-secret-shared-value/.value
      version: ""
    secretKey: my-secret-shared-value
    secretStoreName: ${SECRET_STORE_ID}
  secretName: shared-secrets
EOF

Expected output:

secretmapping.humanitec.io/secretmapping-shared-secrets created

Check the status of the SecretMapping resource:

kubectl get secretmappings -n ${APP_ID} secretmapping-shared-secrets \
  -o jsonpath='{.status.conditions[0]}' | jq

Expected output:

{
  "lastTransitionTime": "2030-11-28T09:24:41Z",
  "message": "Secret applied",
  "reason": "SecretApplied",
  "status": "True",
  "type": "SecretReady"
}

The Operator should have generated a Kubernetes Secret based on the SecretMapping containing the actual secret value. Check the content of that Kubernetes Secret. This step verifies that the Humanitec Operator has read access to the target secret store.

kubectl get secret -n ${APP_ID} shared-secrets \
  -o jsonpath='{.data.my-secret-shared-value}' | base64 -d

Expected output:

super secret Shared Value

Create a Resource custom resource. Its purpose is to write a resource cookie to the default secret store. It uses the convenience config Resource Type and the Template Driver, so no physical resource needs to be created. Its matching criteria match the Application you created earlier.

cat << EOF | kubectl apply -n ${APP_ID} -f - 
apiVersion: humanitec.io/v1alpha1
kind: Resource
metadata:
  name: demo-resource
spec:
  dependsOn: []
  driver: builtin://template
  driverInputs:
    driver:
      values:
        templates:
          cookie: my demo resource cookie
  driverType: humanitec/template
  guresid: demo-resource-cookie
  id: demo-resource-cookie
  type: config
EOF

Expected output:

resource.humanitec.io/demo-resource created

Check the status of the Resource custom resource:

kubectl get resource -n ${APP_ID} demo-resource \
  -o jsonpath='{.status.conditions[0]}' | jq

Expected output:

{
  "lastTransitionTime": "2030-11-28T09:58:50Z",
  "message": "Resource provisioned",
  "reason": "ResourceProvisioned",
  "status": "True",
  "type": "Provisioned"
}

Check for the existence of a new resource cookie in the secret store. You should see the value of the new secret, proving that the Humanitec Operator has write access to the default secret store.

If your default secret store is different from the one defined via SECRET_STORE_ID, make sure to target that secret store in the commands below to see the resource cookie.

aws secretsmanager get-secret-value \
  --secret-id resources/active/demo-resource-cookie/cookies/humanitec/template \
  --region ${SECRETS_MANAGER_REGION} \
  --query SecretString \
  --output text | base64 -d | tr -d '"'

az keyvault secret show \
  --name resources--active--demo-resource-cookie--cookies--humanitec--template \
  --vault-name ${KEY_VAULT_NAME} \
  --query "value" | tr -d '"' | base64 -d | tr -d '"'

gcloud secrets versions access 1 \
  --secret=resources--active--demo-resource-cookie--cookies--humanitec--template | \
  base64 -d | tr -d '"'

vault kv get -mount=${SECRET_PATH} \
  -field=value resources/active/demo-resource-cookie/cookies/humanitec/template | \
  base64 -d | tr -d '"'

Expected output:

my demo resource cookie

Cleanup

Remove all of the objects you created during the test run.

Delete the namespace, which will also delete all of the objects it contains.

kubectl delete namespace ${APP_ID}

Perform the cleanup steps for all tests.

Test: End-to-end

This test lets you deploy an Application via the Platform Orchestrator. The Application uses secret references in two ways: through a Resource Definition using the Echo Driver, and through a Shared Value on the Application level. The Humanitec Operator must resolve all of these secret references to their actual values, reading them from the secret store.

A specific Resource makes the Operator write a resource cookie to the default secret store.

Objectives

Verify the Workload deployment functionality of the Platform Orchestrator / Humanitec Operator toolchain on the target cluster.
Verify that the Operator can inject secrets from the secret store into the deployed Workload.
Verify read access on the selected secret store for the Humanitec Operator.
Verify write access on the default secret store for the Humanitec Operator.

Prerequisites

All of the prerequisites for all tests.
You have the humctl Humanitec CLI installed.
At least one secret store is configured in both Platform Orchestrator and Humanitec Operator, e.g. HashiCorp Vault. Select one target secret store to use in the tests if you have more than one.
You have performed the setup steps for all tests.
You have a Resource Definition for the target Kubernetes cluster set up in the Platform Orchestrator.

Test Steps

Prepare your local environment. Replace all <placeholder values> with the values of your setup.

export HUMANITEC_ORG=<my-org-id>
export HUMANITEC_TOKEN=<my-API-token-for-my-org>
export K8S_RESDEF=<my-target-k8s-cluster-resource-definition-id>

Expected result: all required environment variables are set.

Create an Application in the Platform Orchestrator.

humctl create application ${APP_ID} -n ${APP_NAME}

Expected output similar to:

id      name
my-app  My-App

curl "https://api.humanitec.io/orgs/${HUMANITEC_ORG}/apps" \
  -X POST \
  -H "Authorization: Bearer ${HUMANITEC_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
  "id": "'${APP_ID}'",
  "name": "'${APP_NAME}'"
}'

Expected output similar to:

{"id":"my-app","name":"My-App","org_id":"my-org","created_at":"2030-11-27T09:25:59.856902846Z","created_by":"s-d3e60a0e-8a53-40e9-a686-34548b7e06f0","envs":[{"id":"development","name":"Development","type":"development"}]}

Add a secret Shared Value to the Application. It will read its real value from the secret store using a secret reference.

humctl api post /orgs/${HUMANITEC_ORG}/apps/${APP_ID}/values \
  -d '{
  "description": "My secret Application Shared Value",
  "is_secret": true,
  "key": "my-secret-shared-value",
  "secret_ref": {
    "store": "'${SECRET_STORE_ID}'",
    "ref": "my-secret-shared-value/.value"
  }
}'

Expected output similar to:

{
  "created_at": "2030-11-27T09:27:07.311583767Z",
  "description": "My secret Application Shared Value",
  "is_secret": true,
  "key": "my-secret-shared-value",
  "secret_key": "my-secret-shared-value/.value",
  "secret_store_id": "my-vault-store",
  "secret_version": null,
  "source": "app",
  "updated_at": "2030-11-27T09:27:07.311583767Z",
  "value": ""
}

curl https://api.humanitec.io/orgs/${HUMANITEC_ORG}/apps/${APP_ID}/values \
  -X POST \
  -H "Authorization: Bearer ${HUMANITEC_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
  "description": "My secret Application Shared Value",
  "is_secret": true,
  "key": "my-secret-shared-value",
  "secret_ref": {
    "store": "'${SECRET_STORE_ID}'",
    "ref": "my-secret-shared-value/.value"
  }
}'

Expected output similar to:

{"created_at":"2030-11-27T09:26:07.716583767Z","description":"My secret Application Shared Value","is_secret":true,"key":"my-secret-shared-value","secret_key":"my-secret-shared-value/.value","secret_store_id":"my-vault-store","secret_version":null,"source":"app","updated_at":"2030-11-27T09:26:07.716583767Z","value":""}

humctl api post /orgs/${HUMANITEC_ORG}/apps/${APP_ID}/values \
  -d '{
  "description": "My secret Application Shared Value",
  "is_secret": true,
  "key": "my-secret-shared-value",
  "secret_ref": {
    "store": "'${SECRET_STORE_ID}'",
    "ref": "my-secret-shared-value"
  }
}'

Expected output similar to:

{
  "created_at": "2030-11-27T09:27:07.311583767Z",
  "description": "My secret Application Shared Value",
  "is_secret": true,
  "key": "my-secret-shared-value",
  "secret_key": "my-secret-shared-value",
  "secret_store_id": "my-store",
  "secret_version": null,
  "source": "app",
  "updated_at": "2030-11-27T09:27:07.311583767Z",
  "value": ""
}

curl https://api.humanitec.io/orgs/${HUMANITEC_ORG}/apps/${APP_ID}/values \
  -X POST \
  -H "Authorization: Bearer ${HUMANITEC_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
  "description": "My secret Application Shared Value",
  "is_secret": true,
  "key": "my-secret-shared-value",
  "secret_ref": {
    "store": "'${SECRET_STORE_ID}'",
    "ref": "my-secret-shared-value"
  }
}'

Expected output similar to:

{"created_at":"2030-11-27T09:26:07.716583767Z","description":"My secret Application Shared Value","is_secret":true,"key":"my-secret-shared-value","secret_key":"my-secret-shared-value","secret_store_id":"my-store","secret_version":null,"source":"app","updated_at":"2030-11-27T09:26:07.716583767Z","value":""}

Add matching criteria to the Resource Definition of the target cluster to match this Application. The command captures and outputs the ID of the new matching criteria in the variable CRITERIA_ID for later use.

export CRITERIA_ID=$(humctl api post /orgs/${HUMANITEC_ORG}/resources/defs/${K8S_RESDEF}/criteria \
  -d '{
    "env_type": "development",
    "app_id": "'${APP_ID}'"
  }' | jq --raw-output '.id' )
echo $CRITERIA_ID

export CRITERIA_ID=$(curl -s "https://api.humanitec.io/orgs/${HUMANITEC_ORG}/resources/defs/${K8S_RESDEF}/criteria" \
  -X POST \
  -H "Authorization: Bearer ${HUMANITEC_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "env_type": "development",
    "app_id": "'${APP_ID}'"
  }' | jq --raw-output '.id' )
echo $CRITERIA_ID

Expected output similar to:

d159797fd0709872

Create this Resource Definition. Its purpose is to read values from the secret store using secret references. It uses the Echo Driver to simulate a MySQL resource, and its matching criteria match the Application you created earlier.

Create a file defining the Resource Definition:

cat << EOF > echo-mysql.yaml
apiVersion: entity.humanitec.io/v1b1
kind: Definition
metadata:
  id: echo-mysql
entity:
  name: Echo MySQL
  type: mysql
  driver_type: humanitec/echo
  driver_inputs:
    secret_refs:
      password:
        ref: mysql-password/.value
        store: ${SECRET_STORE_ID}
      username:
        ref: mysql-username/.value
        store: ${SECRET_STORE_ID}
    values:
      host: products.mysql.example.com
      name: example-db
      port: 3306
  criteria:
  - app_id: ${APP_ID}
EOF

Use the humctl create command to create the Resource Definition:

humctl create -f echo-mysql.yaml
rm echo-mysql.yaml

Expected output:

id              type    driver_type     driver_account
echo-mysql      mysql   humanitec/echo  -

curl https://api.humanitec.io/orgs/${HUMANITEC_ORG}/resources/defs \
  -H "Authorization: Bearer ${HUMANITEC_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
  "id": "echo-mysql",
  "name": "Echo MySQL",
  "type": "mysql",
  "driver_type": "humanitec/echo",
  "driver_inputs": {
    "values": {
      "name": "example-db",
      "host": "products.mysql.example.com",
      "port": 3306
    },
    "secret_refs": {
      "username": {
        "store": "'${SECRET_STORE_ID}'",
        "ref": "mysql-username/.value"
      },
      "password": {
        "store": "'${SECRET_STORE_ID}'",
        "ref": "mysql-password/.value"
      }
    }
  },
  "criteria": [
    {
      "app_id":"'${APP_ID}'"
    }
  ]
}'

Expected output similar to:

{"org_id":"my-org","id":"echo-mysql","name":"Echo MySQL","type":"mysql","driver_type":"humanitec/echo","driver_inputs":{"values":{"host":"products.mysql.example.com","name":"example-db","port":3306},"secret_refs":{"password":{"store":"my-vault-store","ref":"mysql-password/.value"},"username":{"store":"my-vault-store","ref":"mysql-username/.value"}}},"created_by":"s-d3e60a0e-8a53-40e9-a676-74548b7e16e0","created_at":"2030-11-27T09:26:25.796342924Z","criteria":[{"app_id":"my-app","class":"default","id":"600ee00742896107"}]}

Create a file defining the Resource Definition:

cat << EOF > echo-mysql.yaml
apiVersion: entity.humanitec.io/v1b1
kind: Definition
metadata:
  id: echo-mysql
entity:
  name: Echo MySQL
  type: mysql
  driver_type: humanitec/echo
  driver_inputs:
    secret_refs:
      password:
        ref: mysql-password
        store: ${SECRET_STORE_ID}
      username:
        ref: mysql-username
        store: ${SECRET_STORE_ID}
    values:
      host: products.mysql.example.com
      name: example-db
      port: 3306
  criteria:
  - app_id: ${APP_ID}
EOF

Use the humctl create command to create the Resource Definition:

humctl create -f echo-mysql.yaml
rm echo-mysql.yaml

Expected output:

id              type    driver_type     driver_account
echo-mysql      mysql   humanitec/echo  -

curl https://api.humanitec.io/orgs/${HUMANITEC_ORG}/resources/defs \
  -H "Authorization: Bearer ${HUMANITEC_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
  "id": "echo-mysql",
  "name": "Echo MySQL",
  "type": "mysql",
  "driver_type": "humanitec/echo",
  "driver_inputs": {
    "values": {
      "name": "example-db",
      "host": "products.mysql.example.com",
      "port": 3306
    },
    "secret_refs": {
      "username": {
        "store": "'${SECRET_STORE_ID}'",
        "ref": "mysql-username"
      },
      "password": {
        "store": "'${SECRET_STORE_ID}'",
        "ref": "mysql-password"
      }
    }
  },
  "criteria": [
    {
      "app_id":"'${APP_ID}'"
    }
  ]
}'

Expected output similar to:

{"org_id":"my-org","id":"echo-mysql","name":"Echo MySQL","type":"mysql","driver_type":"humanitec/echo","driver_inputs":{"values":{"host":"products.mysql.example.com","name":"example-db","port":3306},"secret_refs":{"password":{"store":"my-store","ref":"mysql-password"},"username":{"store":"my-asm","ref":"mysql-username"}}},"created_by":"s-d3e60a0e-8a53-40e9-a676-74548b7e16e0","created_at":"2030-11-27T09:26:25.796342924Z","criteria":[{"app_id":"my-app","class":"default","id":"600ee00742896107"}]}

Create this Resource Definition. Its purpose is to write a resource cookie to the default secret store. It uses the convenience config Resource Type and the Template Driver, so no physical resource needs to be created. Its matching criteria match the Application you created earlier.

Create a file defining the Resource Definition:

cat << EOF > cookie-config.yaml
apiVersion: entity.humanitec.io/v1b1
kind: Definition
metadata:
  id: cookie-config
entity:
  criteria:
  - app_id: ${APP_ID}
    class: default
  driver_inputs:
    values:
      templates:
        cookie: my demo resource cookie
  driver_type: humanitec/template
  name: Cookie config
  type: config
EOF

Use the humctl create command to create the Resource Definition:

humctl create -f cookie-config.yaml
rm cookie-config.yaml

curl https://api.humanitec.io/orgs/${HUMANITEC_ORG}/resources/defs \
  -H "Authorization: Bearer ${HUMANITEC_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
  "id": "cookie-config",
  "name": "Cookie config",
  "type": "config",
  "driver_type": "humanitec/template",
  "driver_inputs": {
    "values": {
      "templates": {
        "cookie": "my demo resource cookie"
      }
    }
  },
  "criteria": [
    {
      "app_id":"'${APP_ID}'"
    }
  ]
}'

Expected output similar to:

{"org_id":"my-org","id":"cookie-config","name":"Cookie config","type":"config","driver_type":"humanitec/template","driver_inputs":{"values":{"templates":{"cookie":"my demo resource cookie"}}},"created_by":"s-d3e60a0e-8a53-40e9-a666-74548b7e06e0","created_at":"2030-11-27T17:07:07.792263908Z","criteria":[{"app_id":"my-app","class":"default","id":"d344a2d9a45ec68e"}]}

Create this Score file for the Application Workload.
- It requests a Resource of type environment. This Resource provides access to the Application’s Shared Value which is injected into an environment variable.
- It requests a Resource of type mysql and injects two of its outputs into a simple container which logs its environment variables. This Resource request will be matched to the Resource Definition of type mysql you created earlier.
- It requests a Resource of type config. This Resource request will be matched to the Resource Definition of type config you created earlier.
- You may replace the image with another suitable image you might want to use as long as it has the printenv command. Provide the fully qualified path to an internal image registry as required.

cat << EOF > score.yaml
apiVersion: score.dev/v1b1
metadata:
  name: ${APP_ID}
containers:
  hello-world:
    image: busybox
    command: ["/bin/sh"]
    args: ["-c", "while true; do printenv && sleep 60; done"]
    variables:
      SECRET_SHARED_VALUE: "\${resources.env.my-secret-shared-value}"
      MYSQL_USERNAME: "\${resources.my-sql.username}"
      MYSQL_PASSWORD: "\${resources.my-sql.password}"
resources:
  env:
    type: environment
  my-sql:
    type: mysql
  my-config:
    type: config
EOF

Expected result: the Score file score.yaml has been created.

Deploy the Application.

humctl score deploy \
  --token $HUMANITEC_TOKEN \
  --org $HUMANITEC_ORG \
  --app ${APP_ID} \
  --env development \
  -f score.yaml

Expected output similar to:

Delta ID                                    Pipeline ID     Pipeline Run ID                         Pipeline Run Status
4d0ee72d6eca8a84a4cffbbf883ef3e707542973    default         41d90b0b-5613-49e5-b2e1-843620c407a6    queued

Now observe the resulting objects on the target Kubernetes cluster. A workload custom resource should have been created in a new namespace. First obtain the namespace.

export APP_NAMESPACE=$(kubectl get workload -A \
  -l app.humanitec.io/app-id=${APP_ID} \
  -o custom-columns=NAMESPACE:metadata.namespace \
  --no-headers)
echo $APP_NAMESPACE

Expected output similar to:

a61741cc-5dec-4eb4-b2cd-80c1343d6031

Check the status of the custom resource of type Workload in the Application namespace and its status.

kubectl get workload -n $APP_NAMESPACE \
  -o jsonpath='{.items[0].status.conditions[0]}' | jq

Expected result similar to:

{
  "lastTransitionTime": "2030-11-27T09:27:01Z",
  "message": "Workload rolled out",
  "reason": "WorkloadRolledOut",
  "status": "True",
  "type": "Ready"
}

You should see "message": "Workload rolled out" and "status": "True" in the output.

Check for the custom resources of type Resource in the Application namespace and their status.

kubectl get resource -n $APP_NAMESPACE \
  -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,ID:.spec.id,PROVISIONED:.status.provisioned

Expected result similar to:

NAMESPACE                              NAME                                                ID                                   PROVISIONED
a61741cc-5dec-4eb4-b2cd-80c1343d6031   resource-39e2aae3561cba6f6cf332363e76b9b3be798378   modules.my-app                       True
a61741cc-5dec-4eb4-b2cd-80c1343d6031   resource-4f33f1e60e014c4b837826464e71d95d57f795e3   modules.my-app.externals.my-config   True
a61741cc-5dec-4eb4-b2cd-80c1343d6031   resource-5f13f05f00b6ee3888ce0532ddbd4fc45ff6ca94   base-env                             True
a61741cc-5dec-4eb4-b2cd-80c1343d6031   resource-c37c6e88f32902c930c3cc46c75af456db507692   modules.my-app.externals.my-sql      True

You should see four resources with a PROVISIONED status of True in the output.

Check for the custom resource of type SecretMapping in the Application namespace and its status.

kubectl get secretmapping -n $APP_NAMESPACE -o jsonpath='{.items[0].status.conditions[0]}' | jq

Expected output similar to:

{
  "deploymentID": "179b70b68f7f5f3b",
  "lastTransitionTime": "2030-11-27T09:26:58Z",
  "message": "Secret applied",
  "reason": "SecretApplied",
  "status": "True",
  "type": "SecretReady"
}

You should see "message": "Secret applied" and "status": "True" in the output.

Get the ID of the Active Resource that was created for the config type Resource Definition.

export CONFIG_RESOURCE_ID=$(kubectl get resource -n $APP_NAMESPACE \
  -o jsonpath='{.items[?(@.metadata.annotations.app\.humanitec\.io/definition-id == "cookie-config")].metadata.name}' | sed 's/resource-//')
echo $CONFIG_RESOURCE_ID

Expected output similar to:

5f33b1e60e014c4c837826464e81d95d57f795e3

Check for the existence of a new resource cookie in the secret store. You should see the value of the new secret, proving that the Humanitec Operator has write access to the default secret store.

If your default secret store is different from the one defined via SECRET_STORE_ID, make sure to target that secret store in the commands below to see the resource cookie.

aws secretsmanager get-secret-value \
  --secret-id resources/active/${CONFIG_RESOURCE_ID}/cookies/humanitec/template \
  --region ${SECRETS_MANAGER_REGION} \
  --query SecretString \
  --output text | base64 -d | tr -d '"'

az keyvault secret show \
  --name resources--active--${CONFIG_RESOURCE_ID}--cookies--humanitec--template \
  --vault-name ${KEY_VAULT_NAME} \
  --query "value" | tr -d '"' | base64 -d | tr -d '"'

gcloud secrets versions access 1 \
  --secret=resources--active--${CONFIG_RESOURCE_ID}--cookies--humanitec--template | \
  base64 -d | tr -d '"'

vault kv get -mount=${SECRET_PATH} -field=value \
  resources/active/${CONFIG_RESOURCE_ID}/cookies/humanitec/template | \
  base64 -d | tr -d '"'

Expected output:

my demo resource cookie

Check for Kubernetes Secrets in the Application namespace.

kubectl get secret -n $APP_NAMESPACE --show-labels

Expected output similar to:

...
resource-secret-c37c6e88f32902c930c3cc46c75af456db507692   Opaque   2      5m   humanitec.io/created-by=operator
shared-secrets                                             Opaque   1      5m   humanitec.io/created-by=operator
...

In the list of secrets returned, you should see those two:

The “resource-secret-*” Secret with two DATA items. This Secret contains the secret values for the MySQL Resource.
The “shared-secrets” Secret with one DATA item. This Secret contains the secret value of the Application Shared Value.

Finally, check the log output of the running container for correct values of the environment variables SECRET_SHARED_VALUE, MYSQL_USERNAME, and MYSQL_PASSWORD. It should show all the values you entered into the secret store secrets earlier, proving that mapping and reading secrets from the store works as expected.

kubectl logs --tail=50 -n \
  $(kubectl get pods -n $APP_NAMESPACE \
  --no-headers \
  -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name) | \
  grep -E -e '^SECRET_SHARED_VALUE|^MYSQL_USERNAME|^MYSQL_PASSWORD'

Expected output:

...
MYSQL_USERNAME=super secret username
MYSQL_PASSWORD=super secret password
SECRET_SHARED_VALUE=super secret Shared Value
...

Cleanup

Remove all of the objects you created during the test run.

Delete the Application from the Platform Orchestrator. This will make the Humanitec Orchestrator remove all associated Kubernetes objects.

humctl delete application ${APP_ID}

curl "https://api.humanitec.io/orgs/${HUMANITEC_ORG}/apps/${APP_ID}" \
  -X DELETE \
  -H "Authorization: Bearer ${HUMANITEC_TOKEN}"

Remove the Score file.

rm score.yaml

Remove the Resource Definitions.

humctl delete resource-definition echo-mysql
humctl delete resource-definition cookie-config

curl https://api.humanitec.io/orgs/${HUMANITEC_ORG}/resources/defs/echo-mysql \
  -X DELETE \
  -H "Authorization: Bearer ${HUMANITEC_TOKEN}"
curl https://api.humanitec.io/orgs/${HUMANITEC_ORG}/resources/defs/cookie-config \
  -X DELETE \
  -H "Authorization: Bearer ${HUMANITEC_TOKEN}"

Remove the matching criteria from the cluster Resource Definition using the CRITERIA_ID captured on creation.

humctl api delete /orgs/${HUMANITEC_ORG}/resources/defs/${K8S_RESDEF}/criteria/${CRITERIA_ID}

curl "https://api.humanitec.io/orgs/${HUMANITEC_ORG}/resources/defs/${K8S_RESDEF}/criteria/${CRITERIA_ID}" \
  -X DELETE \
  -H "Authorization: Bearer ${HUMANITEC_TOKEN}"

Perform the cleanup steps for all tests.

Cleanup (all tests)

Perform these common cleanup steps after executing any of the tests above.

Delete the demo secrets from your secret store.

aws secretsmanager delete-secret \
  --secret-id mysql-username --region ${SECRETS_MANAGER_REGION} \
  --force-delete-without-recovery
aws secretsmanager delete-secret \
  --secret-id mysql-password --region ${SECRETS_MANAGER_REGION} \
  --force-delete-without-recovery
aws secretsmanager delete-secret \
  --secret-id my-secret-shared-value --region ${SECRETS_MANAGER_REGION} \
  --force-delete-without-recovery

az keyvault secret delete -n mysql-username --vault-name ${KEY_VAULT_NAME}
az keyvault secret delete -n mysql-password --vault-name ${KEY_VAULT_NAME}
az keyvault secret delete -n my-secret-shared-value --vault-name ${KEY_VAULT_NAME}

gcloud secrets delete mysql-username --project ${GSM_PROJECT_ID} --quiet
gcloud secrets delete mysql-password --project ${GSM_PROJECT_ID} --quiet
gcloud secrets delete my-secret-shared-value --project ${GSM_PROJECT_ID} --quiet

vault kv metadata delete ${SECRET_PATH}/mysql-username
vault kv metadata delete ${SECRET_PATH}/mysql-password
vault kv metadata delete ${SECRET_PATH}/my-secret-shared-value

Troubleshooting

The test cases contain a number of verification steps to be executed along the way. If at some point the expected result should not be met, utilize these sources to identify the root cause:

Check the status of the custom resource objects (Resource, SecretMapping, Workload) on the Kubernetes cluster that is currently being worked on. The message field will provide helpful information.

kubectl get resource,secretmapping,workload -A \
-o custom-columns=KIND:kind,NAMESPACE:.metadata.namespace,NAME:.metadata.name,STATUS:.status.conditions[0].status,MESSAGE:.status.conditions[0].message

Check the Humanitec Operator logs for errors:

kubectl logs -n humanitec-operator-system \
  -l app.kubernetes.io/instance=humanitec-operator \
  --tail=-1

Testing

Overview #

Prerequisites (all tests) #

Setup (all tests) #

Test: Air-gapped #

Objectives #

Prerequisites #

Test Steps #

Cleanup #

Test: End-to-end #

Objectives #

Prerequisites #

Test Steps #

Cleanup #

Cleanup (all tests) #

Troubleshooting #

Overview

Prerequisites (all tests)

Setup (all tests)

Test: Air-gapped

Objectives

Prerequisites

Test Steps

Cleanup

Test: End-to-end

Objectives

Prerequisites

Test Steps

Cleanup

Cleanup (all tests)

Troubleshooting