Terraform

What is the Terraform Driver?

The Terraform Driver is a fully customizable Driver offered by the Humanitec Platform Orchestrator, allowing you to provision any Resource Type your Workloads depend on. It does that using Terraform and the Terraform providers of your choice.

Platform teams usually set up the Driver for the developers’ requested Resource Types. The setup involves creating Resource Definitions using the Humanitec Terraform provider, and providing the corresponding Terraform code to provision those resources. The Driver will execute the code (terraform apply) at deployment time using standard Terraform just like in any other CI/CD setting.

Why use the Terraform Driver?

Using Terraform and the Terraform Driver to provision resources has a number of advantages over other Humanitec built-in drivers:

  • Declarative: It is declarative, allowing true Infrastructure-as-Code (IaC) with all its associated benefits, like diffs, version control, and familiar developer workflows.
  • Adaptive: Terraform is widely adopted and offers a vast range of providers, covering all major public clouds as well as many 3rd party offerings.
  • Integrate: It provides an easy way to do custom code in Humanitec.
  • Control: Since you control the Terraform code, you have full control of how it operates. You can test the code independently before handing it to the Driver for execution.

How it works

The Terraform Driver and its connecting components can be positioned using the planes of the Humanitec reference architectures as a backdrop.

The Terraform Driver in the Reference Architecture

Please note that this diagram shows just one possible setup. There are several options on how to provide Terraform code (see Inputs and Outputs), where to host the Runner, and where to store state. Detailed documentation on these topics is currently being prepared.

The Platform Orchestrator performs these steps when executing a Deployment into an Application Environment, which leads to the involving of the Terraform Driver.

  1. The Orchestrator performs resource matching using all relevant Resource Definitions. The matched Resource Definition uses the Terraform Driver.
  2. The Orchestrator passes the values of all the inputs to the Terraform Driver as configured in the Resource Definition.
  3. The Driver instantiates an isolated Runner instance to execute Terraform. The Runner includes the Terraform CLI.
  4. The Runner downloads the Terraform Code from the Git repository that is configured in the Resource Definition.
  5. If the Humanitec SaaS solution is used and the Terraform backend configuration uses local state, the Runner downloads an existing state file from an Orchestrator-managed storage. Otherwise, the configured backend is used as is.
  6. The Runner performs terraform init and terraform apply in the repository directory configured in the Resource Definition. For the apply, any variables and secrets configured in the Resource Definition are passed in as input variables on the command line.
  7. After completion, if the Terraform state was previously downloaded, the Runner uploads the updated Terraform state to the Orchestrator-managed storage.
  8. The Runner finishes and returns a response to the driver.
  9. In particular, the Runner passes any output values of the Terraform module back to the driver, who passes them on to the Orchestrator.
  10. The Driver destroys the Runner instance.
  11. Further processing of the response inside the Orchestrator occurs like with any other driver.

Using Terraform providers

There is no restriction on which Terraform providers can be used. You may need to provide the Driver with credentials to access the target infrastructure accessed by the provider; for example, credentials for an AWS account or Azure service principal.

There must be a network line-of-sight to the target infrastructure from the physical execution environment of the Runner.

Providers generally need credentials to access the target infrastructure. See our credentials example for ways to handle them.

Using Terraform Backends

Terraform state is stored according to the backend configured in your Terraform code.

See our backends example for ways to handle backends.

Matching Terraform Modules to Resource Definitions

The purpose of a Resource Definition is to provision exactly one resource of a particular type thanks to a select driver. In contrast, the Terraform module referenced by the Driver may contain several related resources of any type, which may go beyond the type of the Resource Definition.

While this will technically work and all resources will be properly provisioned, it may make sense to review and potentially adjust the module structure to reap the full benefits of the Platform Orchestrator’s dynamic configuration management. This involves:

  • Ideally creating a 1:1 matching of Resource Definitions to Terraform modules (“molecular Terraform”)
  • Providing the proper module output values in case you want to reference them as Dependent Resources in your workload definitions

In doing so, you will gain maximum reusability of your code and achieve standardization intentional across applications and teams. Because the Resource Definition matches the provisioned reality, the Platform Orchestrator can act as a source of truth regarding active resources in an Application Environment.

Terraform Driver reference

This Driver runs Terraform to provision resources. The Terraform definition can be provided in-line or reference a Terraform module in a Git repository.

Running the Terraform Runner in a target cluster

The Terraform Driver can be configured to execute the Terraform scripts as part of a Kubernetes Job execution in a target Kubernetes cluster, instead of in the Humanitec infrastructure.

This mode involves the creation of Kubernetes resources:

  • Some Kubernetes Secrets to cache Terraform scripts, outputs and sensitive data.
  • A Kubernetes Job which executes Terraform.

Running the scripts in a cluster outside of the Humanitec infrastructure requires the configuration of a custom backend as part of the provided Terraform scripts.

Details about how to provide cluster access to the Humanitec Platform Orchestrator, and how to customize the Runner Pod and image source, are specified below.

Properties

Property Description
Resource type Any
Account type aws, aws-role, azure, gcp

Inputs

Values

Name Type Description
runner_mode string It can be managed or custom-kubernetes. Defaults to managed. It determines where the Terraform Runner runs.
append_logs_to_error boolean If set to true, Terraform logs will be appended to error messages returned by the driver. Defaults to false.
credentials_config object [Optional] An object describing how the provider credentials should be available to the Terraform scripts.
files object [Optional] A Map of filenames to their content to create in the directory before terraform is executed.
script string [Optional] An inline terraform definition in HCL format. If specified with source, it works as override.tf
source object [Optional] A Git repository to use for the Terraform definition.
variables object A Map of variable names that are used as inputs to the Terraform definition and their values.
runner object [Optional] Required if runner_mode is custom-kubernetes, specifies the values of the cluster where the Terraform Runner will run.
manifests_output string [Optional] The name of a Terraform output that contains a list of Manifest Location objects in JSON format. These will be returned as the manifests property in the Driver outputs. See Generating manifests for more details.

At least one of source or script must be specified.

credentials_config object

The credentials_config object describes how the provider credentials should be available to the Terraform scripts.

Property Type Description
environment object Map whose keys are the environment values expected by the Terraform scripts and value can be flattened credential keys (with . as delimiter) or credentials file path.
If the value is * it means that the whole credentials will be available at the specified environment variable. If the value at the specified key is an object, the environment variable assumes the stringified value of it.
Example: AWS_ACCESS_KEY_ID: aws.accessKeyId
file string File path for the file that will be built from credentials. The file path can’t be absolute or use dots.
Example: credentials.tf
variables object Map whose keys are variables expected by the Terraform scripts and values can be flattened credential keys (with . as delimiter) or credentials file path.
If the value is * it means that the whole credentials will be available at the specified variable.
If the key contains ., it is considered as a sub-field of a variable of type objects (e.g. “SECRET.ID: aws_access_key_id” generates a variable SECRET of type object which a field ID whose value is fetched by creds at path aws_access_key_id).
If the value at the specified key is an object, the variable is supplied to the scripts as an object, otherwise as a string.
Example: access_key: aws.accessKeyId

Source object

The source object defines how the Driver uses Terraform definitions that are stored in Git. In order for the Driver to use the source-based Terraform definitions, the repository must be accessible to the Terraform Runner and credentials must be supplied if necessary.

Property Type Description
path string [Optional] Relative path to the scripts: path/to/scripts.
rev string [Optional] Branch name, tag, or commit SHA, for example, /refs/heads/main
url object Repository URL, for example, github.com:my-org/project.git for SSH or https://github.com/my-org/project.git for HTTPS.
username object [Optional] Username to authenticate. The default is git.

According to the value specified in the rev property, the Terraform Runner fetches the Terraform scripts in the repository at a specific reference:

  • If rev is a git reference (e.g. refs/heads/my-branch, refs/tags/my-tag, my-branch or my-tag), that branch or tag is checked out.
  • If rev is a valid commit SHA, that commit is checked out.
  • If rev is empty, the HEAD of the repository is checked out (generally main or master).

This means that if rev is not a commit SHA, we can’t ensure that the scripts used to provision a resource match exactly the ones used to delete the same resource as a branch (or a tag) might have been updated compared to when the resource provisioning happened.

Runner object

Available only in custom-kubernetes runner_mode.

The runner object defines the properties the Driver uses to authenticate to the target cluster where the Terraform Runner should run.

Property Type Description
cluster_type string Type of the cluster, must be one of gke, eks, aks or k8s. This property affects the expected structure for cluster property.
cluster object Object which contains the non-sensitive data needed to access the target cluster.
service_account string [Optional] The Service Account the Terraform Runner Job should run with.
account string [Optional] A Humanitec cloud account id. It must be specified if secrets.runner.credentials is empty. Currently only aws and aws-role account types are supported.
namespace string [Optional] The Namespace where the Runner should run. If empty, the Driver assumes the Namespace is humanitec-terraform.
runner_pod_template string [Optional] The Pod Template Spec manifest which defines the Runner Job Pod in the target cluster.

To be able to successfully access the target k8s cluster, the object cluster should include specific fields, according to the value of cluster_type:

A Service Account needs to exist in the target Namespace and it should have been configured to have the following permissions:

  • on resources of kind secrets - APIGroups "", it needs permission to: create, delete, get, list, update and deletecollection.

Runner Pod Template

The Pod Template Spec defined in the runner_pod_template field will be merged to the default one defined by the Terraform Driver as specified below:

spec:
  containers:
  - name: worker
    image: ghcr.io/humanitec/terraform-runner:<version>
    imagePullPolicy: IfNotPresent
    command:
    - /opt/resource-driver-terraform/worker
    resources:
      limits:
        memory: 1Gi
      requests:
        memory: 1Gi
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
      runAsGroup: 2000
      runAsNonRoot: true
      runAsUser: 2000
      seccompProfile:
        type: RuntimeDefault
    env:
    - name: WORKDIR
      value: /home/runneruser/workspace
    ...
    volumeMounts:
    - mountPath: /home/runneruser/workspace/inputs/inputs.json
      name: inputs
      subPath: inputs.json
    ...
  restartPolicy: Never
  serviceAccountName: <runner.service_account>

A common use case is specifying your own registry to use for the Runner image through a runner_pod_template like this:

spec:
  containers:
    - name: worker
      image: <your-private-image>
      imagePullSecrets:
        - name: regcred
      imagePullPolicy: IfNotPresent

As the Pod defined via runner.runner_pod_template is still supposed to work in combination with the Humanitec Terraform Driver and run the Runner image built by Humanitec, not all the fields in the manifest can be overriden:

  • metadata.namespace can’t be overriden as it is already defined at Job level and specified in runner.namespace value.
  • Some properties in the worker container can’t be overriden: args, command, env, envFrom, volumeMounts and workingDir as defined by the Humanitec Terraform Driver.

Secrets

Name Type Description
files object [Optional] A Map of filenames to their content to create in the directory before terraform is executed.
source object [Optional] Credentials for the Git repository.
terraform object [Optional] Secrets to be used by Terraform CLI Configuration. Currently only available with default managed runner.
variables object A Map of variable names that are used as sensitive inputs to the Terraform definition.
runner object [Optional] It can be used only when runner_mode is custom-kubernetes. Specifies the credentials to gain access to the target cluster.

Source object

Credentials to be used to access the Git repository. The choice of credentials depends on the url format.

Property Type Description
password string [Optional] Password or Personal Account Token - for HTTPS.
ssh_key string [Optional] SSH Private key - for connections over SSH.

Runner object

Available only in custom-kubernetes runner_mode.

The runner object defines the credentials the Driver needs to authenticate to the target cluster where the Terraform Runner should run. It can be empty if an account is specified in the runner object.

Property Type Description
credentials object Provider’s credentials object.
agent_url object [Optional] The signed URL produced by the humanitec/agent driver. It is expected to be a reference to the url output of a Agent resource.

The content of the credentials property depends on the value of cluster_type in the runner object:

Terraform object

Secrets to be used by Terraform CLI Configuration.

Property Type Description
tokens object Map of domain - api token to be used to authenticate against private terraform registries.

Notes

Interaction with Humanitec Resources

Resource Types in Humanitec have a specified Resource Output Schema. In order for a resource to be usable by the Platform Orchestrator, the Terraform module must specify output variables that exactly match this schema.

For example, the s3 resource type has the following output schema:

Name Type Description
aws_access_key_id string, secret [Optional] Credentials for the Git repository.
aws_secret_access_key string, secret A Map of variable names that are used as sensitive inputs to the Terraform definition.
bucket string The bucket name.
region string The region the bucket is hosted in.

Therefore, the Terraform module should have outputs defined similar to:

output "region" {
  value = module.aws_s3.s3_bucket_region
}

output "bucket" {
  value = module.aws_s3.s3_bucket_bucket_domain_name
}

output "aws_access_key_id" {
  value     = var.credentials.access_key
  sensitive = true
}

output "aws_secret_access_key" {
  value     = var.credentials.secret_key
  sensitive = true
}

Examples

See the Terraform Driver examples page for a collection of examples.

FAQ

Where is Terraform state stored?

State is stored according to the backend configured in your Terraform code. Use the driver variables and secrets to provide configuration data for your backend.

If you want to access your state, provide a Terraform backend configuration which instructs the runner to store the state in a storage you control, such as s3, gcs, or azurerm.

If a backend configuration is not provided, the state is stored in an isolated secured storage owned by us and you cannot access it. Supplying a local backend configuration is discouraged as it will be ignored and our storage will be used instead.

For a Kubernetes Terraform backend, the runner should be supplied with a Service Account with permission to manipulate K8s Secrets. Specify the required values through the driver variables and secrets.

What happens if my Terraform state is broken?

If the state breaks during Terraform execution by the driver, it depends on where your state is stored.

If you store your state in a self-managed backend (recommended), make sure to set up regular backups to a secondary, secure location. Restore a previous state revision from the backup.

If Humanitec is storing your state, contact Humanitec support to get the state file sent to you. Analyze and fix the state, then have it replaced via support.

Is my Terraform state encrypted?

If you are using your own backend (recommended), the encryption of the selected storage service applies, which we do not control.

If the Orchestrator stores your state, then it is encrypted at rest.

What happens when the credentials for Terraform expire?

Expired credentials either to a Git source or a target infrastructure will lead to a failed Deployment. Renew the credentials and restart the deployment.

Where can I see the Terraform plan?

Right now, you cannot see the plan. We are working on ways to increase the visibility into the Terraform process.

When does the driver destroy resources?

Destruction of the resources (terraform destroy) occurs when the Orchestrator Resource is deprovisioned according to the lifecycle of the resource. In particular, it is not sufficient to remove a resource dependency in order for the resource to be deprovisioned to allow for rollbacks without losing data in stateful resources.

Do I need to split up my Terraform modules?

The Humanitec Platform Orchestrator is all about standardization by design. Fine-grained Terraform modules allow maximum reusability and maximum standardization across applications and teams, making your life a lot easier. We will provide more guidance on how to smoothly adopt existing Terraform code.

How do I deploy a Resource of a type without a matching Resource Type definition?

To provision anything foundational such as networks or Azure Resource Groups use the “base environment” Resource Type (applies to all workloads of an Application) or “workload” Resource Type (applies to a specific workload). You do not need a Resource Type definition in that case.

For any other type of resource, Humanitec currently must add the Resource Definition first. Please contact Humanitec support.

Can I use any Terraform provider in the Terraform registry?

Yes.

Do I need a Terraform Cloud account to use the driver?

No, unless you wish to utilize Terraform’s remote backend. In that case you need to supply configuration data and credentials to access your Terraform cloud workspace.

What Terraform version is the runner using?

For the Humanitec SaaS system we are currently operating the runner with a recent MPL licensed version of Terraform <1.6.0.

For a self-hosted Platform Orchestrator, the runner image is updated along with platform installation following the same version constraint.

What is the public IP address of the SaaS runner?

The runner in the Humanitec SaaS system uses one of the public IPs listed here.

Can I use a Cloud Account configured in my Humanitec organization to provide credentials to the driver?

No.

Top