Terraform

What is the Terraform Driver? #

The Terraform Driver is a fully customizable Driver offered by the Humanitec Platform Orchestrator, allowing you to provision any Resource Type your Workloads depend on. It does that using Terraform and the Terraform providers of your choice.

Platform teams usually set up the Driver for the developers’ requested Resource Types. The setup involves creating Resource Definitions using the Humanitec Terraform provider, and providing the corresponding Terraform code to provision those resources. The Driver will execute the code (terraform apply) at deployment time using standard Terraform just like in any other CI/CD setting.

The Humanitec Terraform Driver is not to be confused with the Humanitec Terraform provider.

The provider can be used to create Humanitec objects such as Resource Definitions or Applications in your Humanitec organization using Terraform.

The driver is used to provision infrastructure resources outside Humanitec such as cloud storage, databases, or networks, using Terraform.

Why use the Terraform Driver? #

Using Terraform and the Terraform Driver to provision resources has a number of advantages over other Humanitec built-in drivers:

• Declarative: It is declarative, allowing true Infrastructure-as-Code (IaC) with all its associated benefits, like diffs, version control, and familiar developer workflows.
• Adaptive: Terraform is widely adopted and offers a vast range of providers, covering all major public clouds as well as many 3rd party offerings.
• Integrate: It provides an easy way to do custom code in Humanitec.
• Control: Since you control the Terraform code, you have full control of how it operates. You can test the code independently before handing it to the Driver for execution.

How it works #

The Terraform Driver and its connecting components can be positioned using the planes of the Humanitec reference architectures as a backdrop.

Please note that this diagram shows just one possible setup. There are several options on how to provide Terraform code (see Inputs and Outputs), where to host the Runner, and where to store state. Detailed documentation on these topics is currently being prepared.

The Platform Orchestrator performs these steps when executing a Deployment into an Application Environment, which leads to the involving of the Terraform Driver.

1. The Orchestrator performs resource matching using all relevant Resource Definitions. The matched Resource Definition uses the Terraform Driver.
2. The Orchestrator passes the values of all the inputs to the Terraform Driver as configured in the Resource Definition.
3. The Driver instantiates an isolated Runner instance to execute Terraform. The Runner includes the Terraform CLI.
4. The Runner downloads the Terraform Code from the Git repository that is configured in the Resource Definition.
5. If the Humanitec SaaS solution is used and the Terraform backend configuration uses local state, the Runner downloads an existing state file from an Orchestrator-managed storage. Otherwise, the configured backend is used as is.
6. The Runner performs terraform init and terraform apply in the repository directory configured in the Resource Definition. For the apply, any variables and secrets configured in the Resource Definition are passed in as input variables on the command line.
7. After completion, if the Terraform state was previously downloaded, the Runner uploads the updated Terraform state to the Orchestrator-managed storage.
8. The Runner finishes and returns a response to the driver.
9. In particular, the Runner passes any output values of the Terraform module back to the driver, who passes them on to the Orchestrator.
10. The Driver destroys the Runner instance.
11. Further processing of the response inside the Orchestrator occurs like with any other driver.

Using Terraform providers #

There is no restriction on which Terraform providers can be used. You may need to provide the Driver with credentials to access the target infrastructure accessed by the provider; for example, credentials for an AWS account or Azure service principal.

There must be a network line-of-sight to the target infrastructure from the physical execution environment of the Runner.

Providers generally need credentials to access the target infrastructure. See our credentials example for ways to handle them.

Using Terraform Backends #

Terraform state is stored according to the backend configured in your Terraform code.

See our backends example for ways to handle backends.

Matching Terraform Modules to Resource Definitions #

The purpose of a Resource Definition is to provision exactly one resource of a particular type thanks to a select driver. In contrast, the Terraform module referenced by the Driver may contain several related resources of any type, which may go beyond the type of the Resource Definition.

While this will technically work and all resources will be properly provisioned, it may make sense to review and potentially adjust the module structure to reap the full benefits of the Platform Orchestrator’s dynamic configuration management. This involves:

• Ideally creating a 1:1 matching of Resource Definitions to Terraform modules (“molecular Terraform”)
• Providing the proper module output values in case you want to reference them as Dependent Resources in your workload definitions

In doing so, you will gain maximum reusability of your code and achieve standardization intentional across applications and teams. Because the Resource Definition matches the provisioned reality, the Platform Orchestrator can act as a source of truth regarding active resources in an Application Environment.

Terraform Driver reference #

This Driver runs Terraform to provision resources. The Terraform definition can be provided in-line or reference a Terraform module in a Git repository.

Running the Terraform Runner in a target cluster #

The Terraform Driver can be configured to execute the Terraform scripts as part of a Kubernetes Job execution in a target Kubernetes cluster, instead of in the Humanitec infrastructure.

This mode involves the creation of Kubernetes resources:

• Some Kubernetes Secrets to cache Terraform scripts, outputs and sensitive data.
• A Kubernetes Job which executes Terraform.

Running the scripts in a cluster outside of the Humanitec infrastructure requires the configuration of a custom backend as part of the provided Terraform scripts.

Details about how to provide cluster access to the Humanitec Platform Orchestrator, and how to customize the Runner Pod and image source, are specified below.

Properties #

Inputs #

Values #

At least one of source or script must be specified.

credentials_config object #

The credentials_config object describes how the provider credentials should be available to the Terraform scripts.

Source object #

The source object defines how the Driver uses Terraform definitions that are stored in Git. In order for the Driver to use the source-based Terraform definitions, the repository must be accessible to the Terraform Runner and credentials must be supplied if necessary.

According to the value specified in the rev property, the Terraform Runner fetches the Terraform scripts in the repository at a specific reference:

• If rev is a git reference (e.g. refs/heads/my-branch, refs/tags/my-tag, my-branch or my-tag), that branch or tag is checked out.
• If rev is a valid commit SHA, that commit is checked out.
• If rev is empty, the HEAD of the repository is checked out (generally main or master).

This means that if rev is not a commit SHA, we can’t ensure that the scripts used to provision a resource match exactly the ones used to delete the same resource as a branch (or a tag) might have been updated compared to when the resource provisioning happened.

Runner object #

Available only in custom-kubernetes runner_mode.

The runner object defines the properties the Driver uses to authenticate to the target cluster where the Terraform Runner should run.

To be able to successfully access the target k8s cluster, the object cluster should include specific fields, according to the value of cluster_type:

• aks cluster type expects the non-secret AKS Cluster values.
• k8s cluster type expects the non-secret Cluster values.
• eks cluster type expects the non-secret EKS Cluster values.
• gke cluster type expects the non-secret GKE Cluster values.

A Service Account needs to exist in the target Namespace and it should have been configured to have the following permissions:

• on resources of kind secrets - APIGroups "", it needs permission to: create, delete, get, list, update and deletecollection.

Runner Pod Template #

The Pod Template Spec defined in the runner_pod_template field will be merged to the default one defined by the Terraform Driver as specified below:

spec:
  containers:
  - name: worker
    image: ghcr.io/humanitec/terraform-runner:<version>
    imagePullPolicy: IfNotPresent
    command:
    - /opt/resource-driver-terraform/worker
    resources:
      limits:
        memory: 1Gi
      requests:
        memory: 1Gi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      runAsGroup: 2000
      runAsNonRoot: true
      runAsUser: 2000
      seccompProfile:
        type: RuntimeDefault
    env:
    - name: WORKDIR
      value: /home/runneruser/workspace
    ...
    volumeMounts:
    - mountPath: /home/runneruser/workspace/inputs/inputs.json
      name: inputs
      subPath: inputs.json
    ...
  restartPolicy: Never
  serviceAccountName: <runner.service_account>

A common use case is specifying your own registry to use for the Runner image through a runner_pod_template like this:

spec:
  containers:
    - name: worker
      image: <your-private-image>
      imagePullSecrets:
        - name: regcred
      imagePullPolicy: IfNotPresent

If you want to host the Runner image in your own registry, use ghcr.io/humanitec/terraform-runner:1.0.2 as the upstream source for greatest stability. We will continuously update the available version information at this place.

This example shows how to supply a customized Pod Spec for the Runner.

As the Pod defined via runner.runner_pod_template is still supposed to work in combination with the Humanitec Terraform Driver and run the Runner image built by Humanitec, not all the fields in the manifest can be overriden:

• metadata.namespace can’t be overriden as it is already defined at Job level and specified in runner.namespace value.
• Some properties in the worker container can’t be overriden: args, command, env, envFrom, volumeMounts and workingDir as defined by the Humanitec Terraform Driver.

Secrets #

Source object #

Credentials to be used to access the Git repository. The choice of credentials depends on the url format.

Runner object #

Available only in custom-kubernetes runner_mode.

The runner object defines the credentials the Driver needs to authenticate to the target cluster where the Terraform Runner should run. It can be empty if an account is specified in the runner object.

The content of the credentials property depends on the value of cluster_type in the runner object:

• aks cluster type expects the AKS Service Principal Secrets.
• k8s cluster type expects the credentials needed to access a Kubernetes cluster using vanilla kubeconfig parameters.
• eks cluster type expects AWS credentials.
• gke cluster type expects a GCP Service Account Key.

Terraform object #

Secrets to be used by Terraform CLI Configuration.

Terraform has a feature allowing bearer tokens for requests to specific domains to be defined via environment variables. The Environment variables start with the prefix TF_TOKEN_ followed by the domain name with all . replaced with _ and all non-ASCII characters converted to their punycode equivalent. See Terraform CLI: Environment Variable Credentials.

The keys of the token object should be the transformed domain names without the TF_TOKEN_ prefix. The value should the bearer token. For example, to provide a token for Terraform Cloud, app.terraform.io, the following entry could be used:

tokens:
  app_terraform_io: my-personal-access-token-for-terraform-cloud

or when using a secret reference:

tokens:
  app_terraform_io:
    store: my-secret-store
    ref: my-personal-access-token-for-terraform-cloud

Notes #

Interaction with Humanitec Resources #

Resource Types in Humanitec have a specified Resource Output Schema. In order for a resource to be usable by the Platform Orchestrator, the Terraform module must specify output variables that exactly match this schema.

For example, the s3 resource type has the following output schema:

Therefore, the Terraform module should have outputs defined similar to:

output "region" {
  value = module.aws_s3.s3_bucket_region
}

output "bucket" {
  value = module.aws_s3.s3_bucket_bucket_domain_name
}

output "aws_access_key_id" {
  value     = var.credentials.access_key
  sensitive = true
}

output "aws_secret_access_key" {
  value     = var.credentials.secret_key
  sensitive = true
}

Examples #

See the Terraform Driver examples page for a collection of examples.

FAQ #

Where is Terraform state stored? #

State is stored according to the backend configured in your Terraform code. Use the driver variables and secrets to provide configuration data for your backend.

If you want to access your state, provide a Terraform backend configuration which instructs the runner to store the state in a storage you control, such as s3, gcs, or azurerm.

If a backend configuration is not provided, the state is stored in an isolated secured storage owned by us and you cannot access it. Supplying a local backend configuration is discouraged as it will be ignored and our storage will be used instead.

For a Kubernetes Terraform backend, the runner should be supplied with a Service Account with permission to manipulate K8s Secrets. Specify the required values through the driver variables and secrets.

What happens if my Terraform state is broken? #

If the state breaks during Terraform execution by the driver, it depends on where your state is stored.

If you store your state in a self-managed backend (recommended), make sure to set up regular backups to a secondary, secure location. Restore a previous state revision from the backup.

If Humanitec is storing your state, contact Humanitec support to get the state file sent to you. Analyze and fix the state, then have it replaced via support.

Is my Terraform state encrypted? #

If you are using your own backend (recommended), the encryption of the selected storage service applies, which we do not control.

If the Orchestrator stores your state, then it is encrypted at rest.

What happens when the credentials for Terraform expire? #

Expired credentials either to a Git source or a target infrastructure will lead to a failed Deployment. Renew the credentials and restart the deployment.

Where can I see the Terraform plan? #

Right now, you cannot see the plan. We are working on ways to increase the visibility into the Terraform process.

When does the driver destroy resources? #

Destruction of the resources (terraform destroy) occurs when the Orchestrator Resource is deprovisioned according to the lifecycle of the resource. In particular, it is not sufficient to remove a resource dependency in order for the resource to be deprovisioned to allow for rollbacks without losing data in stateful resources.

Do I need to split up my Terraform modules? #

The Humanitec Platform Orchestrator is all about standardization by design. Fine-grained Terraform modules allow maximum reusability and maximum standardization across applications and teams, making your life a lot easier. We will provide more guidance on how to smoothly adopt existing Terraform code.

How do I deploy a Resource of a type without a matching Resource Type definition? #

To provision anything foundational such as networks or Azure Resource Groups use the “base environment” Resource Type (applies to all workloads of an Application) or “workload” Resource Type (applies to a specific workload). You do not need a Resource Type definition in that case.

For any other type of resource, Humanitec currently must add the Resource Definition first. Please contact Humanitec support.

Can I use any Terraform provider in the Terraform registry? #

Yes.

Do I need a Terraform Cloud account to use the driver? #

No, unless you wish to utilize Terraform’s remote backend. In that case you need to supply configuration data and credentials to access your Terraform cloud workspace.

What Terraform version is the runner using? #

For the Humanitec SaaS system we are currently operating the runner with a recent MPL licensed version of Terraform <1.6.0.

For a self-hosted Platform Orchestrator, the runner image is updated along with platform installation following the same version constraint.

What is the public IP address of the SaaS runner? #

The runner in the Humanitec SaaS system uses one of the public IPs listed here.

Can I use a Cloud Account configured in my Humanitec organization to provide credentials to the driver? #

No.