kubernetes

The kubernetes state storage type lets a runner maintain Terraform state in Kubernetes secrets.

This state storage type results in the runner using a kubernetes backend (Terraform  / OpenTofu ). It is only supported for Kubernetes-based runner types. The secrets will be located on the same cluster the runner is using.

Follow the steps below to configure a state storage of type kubernetes.

Basic example 

resource "platform-orchestrator_kubernetes_agent_runner" "my_runner" {
  # ...
  runner_configuration = {
    # ...
  }
  state_storage_configuration = {
    type = "kubernetes"
    kubernetes_configuration = {
      namespace = "humanitec-runner"
    }
  }
}

Snippet from a sample runner configuration file runner-config.yaml:

runner_configuration:
  ...
state_storage_configuration:
  type: kubernetes
  namespace: humanitec-runner

Configuration

Set the state storage configuration as part of a runner configuration:

Refer to the resource schema of any runner resource, e.g. the kubernetes_agent_runner in the Terraform  or OpenTofu  Kubernetes runner documentation.

runner_configuration:
  ...
state_storage_configuration:
  # Always set the type to "kubernetes" for this storage type
  type: kubernetes
  # Namespace to use for state storage. The runner will create Kubernetes secrets holding state here
  namespace: humanitec-runner

Before you begin

You will need the following resources and permissions:

  • The kubectl CLI  installed and the current context set to target the Kubernetes cluster and using a principal with permission to create Roles and RoleBindings in the runner namespace

We recommend using the same namespace the runner is using to maintain the state storage secrets. All further instructions are based on that recommendation.

Define a namespace 

locals {
  state_storage_namespace = "humanitec-runner"
}

# If the namespace does not exist yet, create it
resource "kubernetes_namespace" "humanitec_runner_state_storage" {
  metadata {
    name = local.state_storage_namespace
  }
}

  1. Define the Kubernetes namespace where to store the state storage secrets
export STATE_STORAGE_NAMESPACE=humanitec-runner
  1. If the namespace does not exist yet, create it
kubectl create namespace ${STATE_STORAGE_NAMESPACE}

Assign permissions to the runner

Prepare and assign the required permissions to the Kubernetes service account used by the runner.

  1. Define the Kubernetes service account used by the runner

locals {
  # ...
  runner_k8s_sa = "humanitec-runner"
}


export RUNNER_K8S_SA=humanitec-runner
  1. Create and assign a Kubernetes Role

Prepare a Role  and RoleBinding . These permissions effective enable the runner to manage secrets in the runner namespace.

resource "kubernetes_role" "humanitec-runner-kubernetes-stage-storage" {
  metadata {
    name      = "humanitec-runner-kubernetes-stage-storage"
    namespace = kubernetes_namespace.humanitec_runner_state_storage.metadata[0].name
  }
  rule {
    api_groups = [""]
    resources  = ["secrets"]
    verbs      = ["create", "get", "list", "update"]
  }
  rule {
    api_groups = ["coordination.k8s.io"]
    resources  = ["leases"]
    verbs      = ["create", "get", "update"]
  }
}

resource "kubernetes_role_binding" "humanitec-runner-kubernetes-stage-storage" {
  metadata {
    name      = "humanitec-runner-kubernetes-stage-storage"
    namespace = kubernetes_namespace.humanitec_runner_state_storage.metadata[0].name
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "Role"
    name      = kubernetes_role.humanitec-runner-kubernetes-stage-storage.metadata[0].name
  }
  subject {
    kind = "ServiceAccount"
    name = local.runner_k8s_sa
  }
}


cat << EOF > kubernetes-state-storage-role-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: humanitec-runner-kubernetes-stage-storage
  namespace: ${STATE_STORAGE_NAMESPACE}
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "get", "list", "update"]
  - apiGroups: ["coordination.k8s.io"]
    resources: ["leases"]
    verbs: ["create", "get", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: humanitec-runner-kubernetes-stage-storage
  namespace: ${STATE_STORAGE_NAMESPACE}
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: humanitec-runner-kubernetes-stage-storage
subjects:
- kind: ServiceAccount
  name: ${RUNNER_K8S_SA}
EOF

Apply the file:

kubectl apply -f kubernetes-state-storage-role-rolebinding.yaml

Configure the state storage for the runner

Prepare the state storage configuration as part of a runner configuration:

resource "platform-orchestrator_kubernetes_agent_runner" "my_runner" {
  # ...
  runner_configuration = {
    # ...
  }
  state_storage_configuration = {
    type = "kubernetes"
    kubernetes_configuration = {
      namespace = kubernetes_namespace.humanitec_runner_state_storage.metadata[0].name
    }
  }
}


cat << EOF > kubernetes-state-storage-config.yaml
state_storage_configuration:
  type: kubernetes
  namespace: ${STATE_STORAGE_NAMESPACE}
EOF

Kubernetes secrets

The runner using the kubernetes state storage will maintain one Kubernetes secret for each environment.

The secrets are named tfstate-default-<environment-uuid>.

This pattern follows the kubernetes backend configuration  setting the {workspace} to default and the {secret_suffix} to the environment uuid.

$ kubectl get secrets -n humanitec-runner
NAME                                                   TYPE     DATA   AGE
tfstate-default-4b41bf32-b337-93a4-ad65-1d565c5e25be   Opaque   1      21h

s3