Installation

This document will guide you through the installation of the Humanitec Agent into your infrastructure, and its registration with the Platform Orchestrator.

Current Helm Chart version: 1.0.0
Current App version: 1.2.0

Prerequisites

To get started you’ll need:

An execution environment for containers inside your own infrastructure. This can be a Kubernetes cluster, Docker on a VM, or a serverless container runtime offering of your cloud provider. Consider these aspects when choosing your environment:
- The Agent container does not have a listener port, it is a long-running background worker. The execution environment has to cope with a container having those properties. An event-driven, request based execution environment relying on a listener port such as Google Cloud Run might not be suited to hosting the Agent.
- While running several replicas of the same Agent instance is supported, avoid a highly dynamic starting and stopping of Agent containers, e.g. through a serverless autoscaling setup, to not disrupt ongoing data transfers. Aim for a stable number of replicas instead.
Management access to the execution environment to install and run a container. For Kubernetes, this means kubectl access. For other types of environments, use an access type at your discretion.
- The execution environment can access the public Humanitec registry to pull the Humanitec Agent OCI image from ghcr.io/humanitec/agent:1.2.0. If that access is restricted, transfer the image to a registry that is accessible for the environment.
- The execution environment has network line-of-sight to the host agent-01.humanitec.io on TCP port 443.
The humctl CLI installed.

Provide a key pair

The Humanitec Agent will need to connect and send requests to the Platform Orchestrator via a websocket tunnel. The Platform Orchestrator verifies the Agent by checking that the requests from the Agent are signed with the Agent’s private key. Part of the setup involves configuring the Agent with the private key and providing the corresponding public key to the Platform Orchestrator.

The following steps show how to register the first public key. To register additional keys later, see the instructions below.

At this time, RSA keys with a bit length ≥ 4096 bits are supported.

The following commands will generate two files humanitec_agent_private_key.pem and humanitec_agent_public_key.pem containing a private and corresponding public key.

# Generate the private key in humanitec_agent_private_key.pem
openssl genrsa -out humanitec_agent_private_key.pem 4096

# Generate public key from the private key in humanitec_agent_public_key.pem
openssl rsa -in humanitec_agent_private_key.pem -outform PEM -pubout -out humanitec_agent_public_key.pem

resource "tls_private_key" "agent_private_key" {
  algorithm = "RSA"
  rsa_bits  = 4096
}

⚠️ Usage of the tls_private_key resource is not recommended for production deployments. For more information, see the tls_private_key Terraform Resource.

Do not set a passphrase.

Register the Humanitec Agent to your Organization

The Humanitec Agent and its public key must then be registered with your Organization via the Agent API. Choose an Agent ID that will be used to uniquely identify the Agent instance within your Organization. The ID may contain lowercase letters, digits, and the “-” symbol.

Set these environment variables:

export HUMANITEC_ORG=<your-humanitec-org-id>
export HUMANITEC_TOKEN=<your-humanitec-api-token>
export AGENT_ID=<your-agent-id>

Prepare a Resource Definition of type: agent. Adjust the criteria according to your Agent usage scenario. The example suggests using this Agent instance across all Environments of type development.

cat << EOF > resource-definition-agent.yaml
apiVersion: entity.humanitec.io/v1b1
kind: Definition
metadata:
  id: ${AGENT_ID}
entity:
  type: agent
  name: ${AGENT_ID}
  driver_type: humanitec/agent
  driver_inputs:
    values:
      id: ${AGENT_ID}
  criteria:
  - env_type: development
    res_id: agent
EOF

Apply the Resource Definition:

humctl apply -f resource-definition-agent.yaml

variable "agent_id" {
  type  = string
}

resource "humanitec_resource_definition" "agent" {
  id   = var.agent_id
  name = var.agent_id
  type = "agent"

  driver_type = "humanitec/agent"
  driver_inputs = {
    values_string = jsonencode({
      id = var.agent_id
    })
  }
}

resource "humanitec_resource_definition_criteria" "agent" {
  resource_definition_id  = humanitec_resource_definition.agent.id
  env_type                = "development"
  res_id                  = "agent"
}

For more information, see the humanitec_resource_definition Terraform Resource.

Other Resources request the Agent by referencing it in the agent_url property of agent-enabled Drivers.

This Resource Definition can then be matched according to the specified matching criteria.

Continue the setup now, and follow the links under “Next steps” at the end of this article to see how it works in practice.

humctl api POST /orgs/${HUMANITEC_ORG}/agents \
  -d '{
  "id": "'${AGENT_ID}'",
  "public_key": '"$(cat humanitec_agent_public_key.pem | jq -sR)"',
  "description": "Demo Agent"
}'

Note: the jq command is used to encode the public key file as a JSON string.

resource "humanitec_agent" "agent" {
  id          = var.agent_id
  description = "Demo Agent"
  public_keys = [
    {
      key = "${var.public_key}"
      # Alternatively, use the tls_private_key resource as shown above. Not recommended for production use.
      # key = "${tls_private_key.agent_private_key.public_key_pem}"
    }
  ]
}

For more information, see the humanitec_agent Terraform Resource.

Install

To run the Humanitec Agent on a Kubernetes cluster, execute the following command to install the required Kubernetes manifests.

Consider these aspects before you execute it, or if you wish to create your own manifests:

If you’re using an image registry other than registry.humanitec.io, change the image value accordingly.
The ConfigMap humanitec-agent-configmap-env delivers environment variables into the Agent Pod which represent the configuration options for the Agent. See Configuration below for details.
The Secret humanitec-agent-secret-private-key holds the private key which is mounted into the Agent Pod through a volume. This Secret supplies the private key at the default path /keys/private_key.pem. See Configuration on how to change the path.
The Agent Pods will get the environment variable CONNECTION_ID set to the Pod name, utilizing the optional Configuration setting to identify the Agent replica. Remove or change this at your discretion.
The private key file must have a mode of 600 and be owned by a user with UID 1000 when made available to the container. This is already taken care of in these manifests.
The command $(cat humanitec_agent_private_key.pem | base64 -w0 ) used for creating the Secret works as-is on a Linux bash. For macOS, omit the -w0 parameter. On other shells, identify the proper parameterization so that a single-line string is created as the Secret data.

kubectl apply -f - <<EOF
---
apiVersion: v1
kind: Namespace
metadata:
  name: humanitec-agent
  labels:
    app.kubernetes.io/name: "humanitec-agent"
    app.kubernetes.io/instance: "humanitec-agent"
---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: humanitec-agent
  name: humanitec-agent-configmap-env
  labels:
    app.kubernetes.io/name: "humanitec-agent"
    app.kubernetes.io/instance: "humanitec-agent"
data:
  ORGS: ${HUMANITEC_ORG}
---
apiVersion: v1
kind: Secret
metadata:
  name: humanitec-agent-secret-private-key
  namespace: humanitec-agent
  labels:
    app.kubernetes.io/name: "humanitec-agent"
    app.kubernetes.io/instance: "humanitec-agent"
data:
  private_key: |
    $(cat humanitec_agent_private_key.pem | base64 -w0 )
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: humanitec-agent
  name: humanitec-agent
  labels:
    app.kubernetes.io/name: "humanitec-agent"
    app.kubernetes.io/instance: "humanitec-agent"
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: "humanitec-agent"
      app.kubernetes.io/instance: "humanitec-agent"
  template:
    metadata:
      labels:
        app.kubernetes.io/name: "humanitec-agent"
        app.kubernetes.io/instance: "humanitec-agent"
    spec:
      securityContext:
        fsGroup: 1000
        fsGroupChangePolicy: Always
        runAsGroup: 1000
        runAsUser: 1000
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: humanitec-agent
          image: "ghcr.io/humanitec/agent:1.2.0"
          resources:
            limits:
              cpu: "0.250"
              memory: 256Mi
            requests:
              cpu: "0.025"
              memory: 64Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            privileged: false
            readOnlyRootFilesystem: true
          env:
          - name: CONNECTION_ID
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          envFrom:
          - configMapRef:
              name: humanitec-agent-configmap-env
          volumeMounts:
          - name: agentmount
            mountPath: "/keys"
            readOnly: true
      volumes:
      - name: agentmount
        projected:
          sources:
          - secret:
              name: humanitec-agent-secret-private-key
              items:
                - key: private_key
                  path: private_key.pem
                  mode: 384 #equivalent of 0600
EOF

You should see this output:

namespace/humanitec-agent created
configmap/humanitec-agent-configmap-env created
secret/humanitec-agent-secret-private-key created
deployment.apps/humanitec-agent created

Install the latest version of the Helm chart:

helm install humanitec-agent \
  oci://ghcr.io/humanitec/charts/humanitec-agent \
  --namespace humanitec-agent \
  --create-namespace \
  --set "humanitec.org=${HUMANITEC_ORG}" \
  --set "humanitec.privateKey=$(cat humanitec_agent_private_key.pem)"

To install a particular version, add a “--version“ parameter to the command.

When using a Helm version prior to 3.8.0, you must manually enable OCI registry support. See this article for instructions.

Verify your installation

Verify the Humanitec Agent’s Pod is running in the target namespace:

kubectl get pods -n humanitec-agent

You should see a Pod like this in a Running state:

NAME                               READY   STATUS    RESTARTS   AGE
humanitec-agent-6f7f89f89f-jshv2   1/1     Running   0          2m

Check the Pod’s logs:

kubectl logs -n humanitec-agent \
  $(kubectl get pods -l app.kubernetes.io/name=humanitec-agent -n humanitec-agent \
   -o custom-columns=NAME:metadata.name --no-headers)

You should see this output:

time=2030-12-15T14:11:56.057Z level=INFO msg="websocket connection established" connectionId=""
time=2030-12-15T14:11:56.057Z level=INFO msg="Connected to websocket tunnel"
time=2030-12-15T14:11:56.057Z level=INFO msg="Starting server" "listening to"=wss://agent-01.humanitec.io/tunnel

If you’re not seeing these results, head to the Troubleshooting section for assistance on how to debug your installation.

Install on Azure Container Instances

To run the Humanitec Agent on an Azure Container Instances service, execute the following commands to install the container.

Consider these aspects before you execute it, or if you wish to create your own installation code:

The command $(cat humanitec_agent_private_key.pem | base64 -w0 ) used for creating the Secret works as-is on a Linux bash. For macOS, omit the -w0 parameter. On other shells, identify the proper parameterization so that a single-line string is created as the Secret data.

Prepare the manifest:

cat << EOF > aci-humanitec-agent.yaml
apiVersion: '2019-12-01'
location: my-location
type: Microsoft.ContainerInstance/containerGroups
name: humanitec-agent
properties:
  containers:
  - name: humanitec-agent
    properties:
      environmentVariables:
        - name: 'ORGS'
          value: '${HUMANITEC_ORG}'
      image: ghcr.io/humanitec/agent:1.2.0
      resources:
        requests:
          cpu: 0.05
          memoryInGB: 0.1
      volumeMounts:
      - mountPath: /keys
        name: agentmount
  osType: Linux
  restartPolicy: Always
  volumes:
  - name: agentmount
    secret:
      private_key.pem: $(cat humanitec_agent_private_key.pem | base64 -w0 )
EOF

Install the manifest:

az container create --resource-group my-resource-group --file aci-humanitec-agent.yaml

Verify your Azure Container Instances installation

Verify the container is running:

az container show -n humanitec-agent -g my-resource-group | jq .containers[0].instanceView.currentState

You should see this output:

{
  "detailStatus": "",
  "exitCode": null,
  "finishTime": null,
  "startTime": "2030-03-11T10:14:14.344000+00:00",
  "state": "Running"
}

Check the container’s logs:

az container logs -n humanitec-agent -g my-resource-group

You should see this output:

time=2030-03-11T10:14:14.445Z level=INFO msg=Starting app=agent-client version=1.2.0 build=2030-02-28T12:47:35Z sha=a1f7b80e73f965e3587dc549d83a0db42b75564f
time=2030-03-11T10:14:14.595Z level=INFO msg="websocket connection estabilished" connectionId=""
time=2030-03-11T10:14:14.595Z level=INFO msg="Connected to websocket tunnel"
time=2030-03-11T10:14:14.596Z level=INFO msg="Starting server" "listening to"=wss://agent-01.humanitec.io/tunnel

Clean up

It’s best to store the public/private key pairs in a safe place for future use, e.g. in a secret store your organization is using. Refer to the product documentation of that store for more details on its use.

Then, remove the generated key pair from your local environment:

rm humanitec_agent_private_key.pem humanitec_agent_public_key.pem

Upgrade your installation

To upgrade your Humanitec Agent installation to the newest version, modify your installation to use the current image version number.

For Kubernetes, you can re-apply the above manifests. This will result in a new Agent Pod being created and the old one terminated.

Configuration

The following environment variables can be used to configure the Agent:

Variable	Description
ORGS	(Required) A comma-separated list of Humanitec Organizations that the Agent can be applied to.
PRIVATE_KEY_FILE	(Optional) The absolute path of the private key. Default: `/keys/private_key.pem`
CONNECTION_ID	(Optional) A string that can be used to identify different replicas of an Agent running under the same private key.
LOG_LEVEL	(Optional) Specifies the log level of the Agent. One of `DEBUG`, `INFO`, `WARN`, or `ERROR`. Default: `INFO`.

Manage Humanitec Agents and keys

Get registered Humanitec Agents

To get currently registered Humanitec Agents for an Organization, use the following command:

humctl api get /orgs/${HUMANITEC_ORG}/agents

Get public keys for a Humanitec Agent

To get currently linked public keys for a Humanitec Agent in an Organization, use the following command:

humctl api get /orgs/${HUMANITEC_ORG}/agents/${AGENT_ID}/keys

De-register a Humanitec Agent

To de-register a Humanitec Agent from the Platform Orchestrator, use the following command. This will also remove all the public keys linked to the Agent:

humctl api delete /orgs/${HUMANITEC_ORG}/agents/${AGENT_ID}

Link an additional public key to a Humanitec Agent

First, create a pair of public/private key files as shown in these instructions.

To link the public key from the file humanitec_agent_public_key.pem to an existing Humanitec Agent, use the following command:

humctl api POST /orgs/${HUMANITEC_ORG}/agents/${AGENT_ID}/keys \
  -d '{
  "public_key": '"$(cat humanitec_agent_public_key.pem | jq -sR)"'
}'

Un-link a public key from a Humanitec Agent

Determine the fingerprint of the public key you wish to un-link by checking the list of registered public keys.

To unlink a particular public key from a Humanitec Agent, use the following command:

humctl api DELETE /orgs/${HUMANITEC_ORG}/agents/${AGENT_ID}/keys/<fingerprint>

Rotate the key pair for a Humanitec Agent

To rotate the key pair used by a registered Humanitec Agent instance, follow these steps:

Generate a new key pair as shown in these instructions.
Link the new public key to the registered Humanitec Agent as shown in these instructions.
Update the Agent configuration to use the new private key.
- The exact steps depend on the execution environment the Agent instance is using.
- For Kubernetes, this means updating the Secret holding the private key.
Restart the Agent to pick up the updated public key.
- For Kubernetes, this means restarting the Agent Pod.
Un-link the old public key from the registered Humanitec Agent as shown in these instructions.

Uninstalling

To uninstall the Humanitec Agent from a Kubernetes cluster, execute the Kubernetes installation manifests, replacing kubectl apply with kubectl delete.

To uninstall the Humantec Agent from a different execution environment, follow the instructions of that environment to uninstall a running container.

Troubleshooting

Troubleshoot connectivity

To analyze whether your Humanitec Agent installation has the required access to the Platform Orchestrator, run a command inside the Agent container using the “netcat” (nc) tool.

Kubernetes:

kubectl exec -n humanitec-agent \
  $(kubectl get pods -l app.kubernetes.io/name=humanitec-agent -n humanitec-agent \
  -o custom-columns=NAME:metadata.name --no-headers) \
  -- nc -v -w 10 agent-01.humanitec.io 443

Docker:

docker exec <agent-container-id> /bin/sh -c "nc -v -w 10 agent-01.humanitec.io 443"

You should see this output if connectivity works:

agent-01.humanitec.io (34.102.177.245:443) open

Debugging network issues is beyond the scope of this document. Consider these aspects in case of connectivity issues:

Is there an egress route from the execution environment of the Humanitec Agent to the endpoint agent-01.humanitec.io?
Does egress traffic have to pass through a firewall or virtual appliance that inspects traffic and might block it on TCP port 443?
If the execution environment is attached to a virtual network/VPC, does that network provide flow logs that might provide insights?
For Kubernetes, are network policies affecting the egress of the Humanitec Agent pod?

HTTP response 401 in Agent logs

If you see this message in the Agent logs:

level=ERROR msg="Can't connect to websocket" err="unable to establish websocket connection: unexpected HTTP response status: 401, { []}"

It means the Agent could connect to the Platform Orchestrator on the network level, but was not authenticated.

Verify that the Agent’s public key is registered for all Organizations configured for the Agent.

Unable to determine cluster version: Bad Gateway

If a Deployment fails that utilizes a Humanitec Agent, and you see this error message in the Platform Orchestrator:

Unable to determine cluster version: Get "https://.35.197.242.20/version": Bad Gateway

It might be caused by one of the following issues:

The target Agent is not running for some reason.
The private key in private_key.pem is signing requests to the Platform Orchestrator with a key that has not been shared or validated yet.

Suggested mitigations are:

Check the status of the target Agent’s container and its logs.
Restart the Agent to re-establish the websocket tunnel with the Platform Orchestrator.
If the private key has been updated but not added to the Agent registered with the Platform Orchestrator, please do it as suggested here.

Next steps

Now that you’ve installed the Humanitec Agent, use it to deploy Workloads to a private Kubernetes cluster.

Installation

Prerequisites #

Provide a key pair #

Register the Humanitec Agent to your Organization #

Install #

Verify your installation #

Install on Azure Container Instances #

Verify your Azure Container Instances installation #

Clean up #

Upgrade your installation #

Configuration #

Manage Humanitec Agents and keys #

Get registered Humanitec Agents #

Get public keys for a Humanitec Agent #

De-register a Humanitec Agent #

Link an additional public key to a Humanitec Agent #

Un-link a public key from a Humanitec Agent #

Rotate the key pair for a Humanitec Agent #

Uninstalling #

Troubleshooting #

Troubleshoot connectivity #

HTTP response 401 in Agent logs #

Unable to determine cluster version: Bad Gateway #

Next steps #

Prerequisites

Provide a key pair

Register the Humanitec Agent to your Organization

Install

Verify your installation

Install on Azure Container Instances

Verify your Azure Container Instances installation

Clean up

Upgrade your installation

Configuration

Manage Humanitec Agents and keys

Get registered Humanitec Agents

Get public keys for a Humanitec Agent

De-register a Humanitec Agent

Link an additional public key to a Humanitec Agent

Un-link a public key from a Humanitec Agent

Rotate the key pair for a Humanitec Agent

Uninstalling

Troubleshooting

Troubleshoot connectivity

HTTP response 401 in Agent logs

Unable to determine cluster version: Bad Gateway

Next steps