Installation

This document will guide you through the installation of the Humanitec Agent into your infrastructure, and its registration with the Platform Orchestrator.

Prerequisites

To get started you’ll need:

  • An execution environment for containers inside your own infrastructure. This can be a Kubernetes cluster, Docker on a VM, or a serverless container runtime offering of your cloud provider. Consider these aspects when choosing your environment:
    • The Agent container does not have a listener port, it is a long-running background worker. The execution environment has to cope with a container having those properties. An event-driven, request based execution environment relying on a listener port such as Google Cloud Run might not be suited to hosting the Agent.
    • While running several replicas of the same Agent instance is supported, avoid a highly dynamic starting and stopping of Agent containers, e.g. through a serverless autoscaling setup, to not disrupt ongoing data transfers. Aim for a stable number of replicas instead.
  • Management access to the execution environment to install and run a container. For Kubernetes, this means kubectl access. For other types of environments, use an access type at your discretion.
    • The execution environment can access the public Humanitec registry to pull the Humanitec Agent OCI image from ghcr.io/humanitec/agent:1.8.2. If that access is restricted, transfer the image to a registry that is accessible for the environment.
  • The execution environment firewall must allow TCP connections to agent-01.humanitec.io on port 443.
  • The humctl CLI installed.

Provide a key pair

The Humanitec Agent will need to connect and send requests to the Platform Orchestrator via a websocket tunnel. The Platform Orchestrator verifies the Agent by checking that the requests from the Agent are signed with the Agent’s private key. Part of the setup involves configuring the Agent with the private key and providing the corresponding public key to the Platform Orchestrator.

The following steps show how to register the first public key. To register additional keys later, see the instructions below .

The following commands will generate two files humanitec_agent_private_key.pem and humanitec_agent_public_key.pem containing a private and corresponding public key.

# Generate the private key in humanitec_agent_private_key.pem
openssl genrsa -out humanitec_agent_private_key.pem 4096

# Generate public key from the private key in humanitec_agent_public_key.pem
openssl rsa -in humanitec_agent_private_key.pem -outform PEM -pubout -out humanitec_agent_public_key.pem


resource "tls_private_key" "agent_private_key" {
  algorithm = "RSA"
  rsa_bits  = 4096
}

Do not set a passphrase.

Register the Humanitec Agent to your Organization

The Humanitec Agent and its public key must then be registered with your Organization via the Agent API. Choose an Agent ID that will be used to uniquely identify the Agent instance within your Organization. The ID may contain lowercase letters, digits, and the “-” symbol.

Set these environment variables:

export HUMANITEC_ORG=<your-humanitec-org-id>
export AGENT_ID=<your-agent-id>

Prepare a Resource Definition of type: agent. Adjust the criteria according to your Agent usage scenario. The example suggests using this Agent instance across all Environments of type development.

cat << EOF > resource-definition-agent.yaml
apiVersion: entity.humanitec.io/v1b1
kind: Definition
metadata:
  id: ${AGENT_ID}
entity:
  type: agent
  name: ${AGENT_ID}
  driver_type: humanitec/agent
  driver_inputs:
    values:
      id: ${AGENT_ID}
  criteria:
  - env_type: development
    res_id: agent
EOF

Apply the Resource Definition:

humctl apply -f resource-definition-agent.yaml


variable "agent_id" {
  type  = string
}

resource "humanitec_resource_definition" "agent" {
  id   = var.agent_id
  name = var.agent_id
  type = "agent"

  driver_type = "humanitec/agent"
  driver_inputs = {
    values_string = jsonencode({
      id = var.agent_id
    })
  }
}

resource "humanitec_resource_definition_criteria" "agent" {
  resource_definition_id  = humanitec_resource_definition.agent.id
  env_type                = "development"
  res_id                  = "agent"
}

Register the Agent and its public key. Adapt the description as appropriate:

humctl api POST /orgs/${HUMANITEC_ORG}/agents \
  -d '{
  "id": "'${AGENT_ID}'",
  "public_key": '"$(cat humanitec_agent_public_key.pem | jq -sR)"',
  "description": "Demo Agent"
}'

Note: the jq command is used to encode the public key file as a JSON string.

resource "humanitec_agent" "agent" {
  id          = var.agent_id
  description = "Demo Agent"
  public_keys = [
    {
      key = "${var.public_key}"
      # Alternatively, use the tls_private_key resource as shown above. Not recommended for production use.
      # key = "${tls_private_key.agent_private_key.public_key_pem}"
    }
  ]
}

Install

To run the Humanitec Agent on a Kubernetes cluster, execute the following command to install the required Kubernetes manifests.

Consider these aspects before you execute it, or if you wish to create your own manifests:

  • If you’re using an image registry other than ghcr.io/humanitec, change the image value accordingly.
  • The ConfigMap humanitec-agent-configmap-env delivers environment variables into the Agent Pod which represent the configuration options for the Agent. See Configuration below for details.
  • The Secret humanitec-agent-secret-private-key holds the private key which is mounted into the Agent Pod through a volume. This Secret supplies the private key at the default path /keys/private_key.pem. See Configuration on how to change the path.
  • The Agent Pods will get the environment variable CONNECTION_ID set to the Pod name, utilizing the optional Configuration setting to identify the Agent replica. Remove or change this at your discretion.
  • The private key file must have a mode of 600 and be owned by a user with UID 1000 when made available to the container. This is already taken care of in these manifests.
  • The command $(cat humanitec_agent_private_key.pem | base64 -w0 ) used for creating the Secret works as-is on a Linux bash. For macOS, omit the -w0 parameter. On other shells, identify the proper parameterization so that a single-line string is created as the Secret data.
kubectl apply -f - <<EOF
---
apiVersion: v1
kind: Namespace
metadata:
  name: humanitec-agent
  labels:
    app.kubernetes.io/name: "humanitec-agent"
    app.kubernetes.io/instance: "humanitec-agent"
---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: humanitec-agent
  name: humanitec-agent-configmap-env
  labels:
    app.kubernetes.io/name: "humanitec-agent"
    app.kubernetes.io/instance: "humanitec-agent"
data:
  ORGS: ${HUMANITEC_ORG}
---
apiVersion: v1
kind: Secret
metadata:
  name: humanitec-agent-secret-private-key
  namespace: humanitec-agent
  labels:
    app.kubernetes.io/name: "humanitec-agent"
    app.kubernetes.io/instance: "humanitec-agent"
data:
  private_key: |
    $(cat humanitec_agent_private_key.pem | base64 -w0 )
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: humanitec-agent
  name: humanitec-agent
  labels:
    app.kubernetes.io/name: "humanitec-agent"
    app.kubernetes.io/instance: "humanitec-agent"
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: "humanitec-agent"
      app.kubernetes.io/instance: "humanitec-agent"
  template:
    metadata:
      labels:
        app.kubernetes.io/name: "humanitec-agent"
        app.kubernetes.io/instance: "humanitec-agent"
    spec:
      securityContext:
        fsGroup: 1000
        fsGroupChangePolicy: Always
        runAsGroup: 1000
        runAsUser: 1000
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: humanitec-agent
          image: "ghcr.io/humanitec/agent:1.8.2"
          resources:
            limits:
              cpu: "0.250"
              memory: 256Mi
            requests:
              cpu: "0.025"
              memory: 64Mi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            privileged: false
            readOnlyRootFilesystem: true
          env:
          - name: CONNECTION_ID
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          envFrom:
          - configMapRef:
              name: humanitec-agent-configmap-env
          volumeMounts:
          - name: agentmount
            mountPath: "/keys"
            readOnly: true
      volumes:
      - name: agentmount
        projected:
          sources:
          - secret:
              name: humanitec-agent-secret-private-key
              items:
                - key: private_key
                  path: private_key.pem
                  mode: 384 #equivalent of 0600
EOF

You should see this output:

namespace/humanitec-agent created
configmap/humanitec-agent-configmap-env created
secret/humanitec-agent-secret-private-key created
deployment.apps/humanitec-agent created

Install the latest version of the Helm chart:

helm install humanitec-agent \
  oci://ghcr.io/humanitec/charts/humanitec-agent \
  --namespace humanitec-agent \
  --create-namespace \
  --set "humanitec.org=${HUMANITEC_ORG}" \
  --set "humanitec.privateKey=$(cat humanitec_agent_private_key.pem)"

To install a particular version, add a “--version” parameter to the command.

The available values for this chart are documented in Agent Helm Chart values .

When using a Helm version prior to 3.8.0, you must manually enable OCI registry support. See this article for instructions.

Verify your installation

Verify the Humanitec Agent’s Pod is running in the target namespace:

kubectl get pods -n humanitec-agent

You should see a Pod like this in a Running state:

NAME                               READY   STATUS    RESTARTS   AGE
humanitec-agent-6f7f89f89f-jshv2   1/1     Running   0          2m

Check the Pod’s logs:

kubectl logs -n humanitec-agent \
  $(kubectl get pods -l app.kubernetes.io/name=humanitec-agent -n humanitec-agent \
   -o custom-columns=NAME:metadata.name --no-headers)

You should see this output:

time=2030-12-15T14:11:56.057Z level=INFO msg="websocket connection established" connectionId=""
time=2030-12-15T14:11:56.057Z level=INFO msg="Connected to websocket tunnel"
time=2030-12-15T14:11:56.057Z level=INFO msg="Starting server" "listening to"=wss://agent-01.humanitec.io/tunnel

If you’re not seeing these results, head to the Troubleshooting section for assistance on how to debug your installation.

Install on Azure Container Instances

To run the Humanitec Agent on an Azure Container Instances service, execute the following commands to install the container.

Consider these aspects before you execute it, or if you wish to create your own installation code:

  • The command $(cat humanitec_agent_private_key.pem | base64 -w0 ) used for creating the Secret works as-is on a Linux bash. For macOS, omit the -w0 parameter. On other shells, identify the proper parameterization so that a single-line string is created as the Secret data.

Prepare the manifest:

cat << EOF > aci-humanitec-agent.yaml
apiVersion: '2019-12-01'
location: my-location
type: Microsoft.ContainerInstance/containerGroups
name: humanitec-agent
properties:
  containers:
  - name: humanitec-agent
    properties:
      environmentVariables:
        - name: 'ORGS'
          value: '${HUMANITEC_ORG}'
      image: ghcr.io/humanitec/agent:1.8.2
      resources:
        requests:
          cpu: 0.05
          memoryInGB: 0.1
      volumeMounts:
      - mountPath: /keys
        name: agentmount
  osType: Linux
  restartPolicy: Always
  volumes:
  - name: agentmount
    secret:
      private_key.pem: $(cat humanitec_agent_private_key.pem | base64 -w0 )
EOF

Install the manifest:

az container create --resource-group my-resource-group --file aci-humanitec-agent.yaml

Verify your Azure Container Instances installation

Verify the container is running:

az container show -n humanitec-agent -g my-resource-group | jq .containers[0].instanceView.currentState

You should see this output:

{
  "detailStatus": "",
  "exitCode": null,
  "finishTime": null,
  "startTime": "2030-03-11T10:14:14.344000+00:00",
  "state": "Running"
}

Check the container’s logs:

az container logs -n humanitec-agent -g my-resource-group

You should see this output:

time=2030-03-11T10:14:14.445Z level=INFO msg=Starting app=agent-client version=1.8.2 build=2030-02-28T12:47:35Z sha=a1f7b80e73f965e3587dc549d83a0db42b75564f
time=2030-03-11T10:14:14.595Z level=INFO msg="websocket connection estabilished" connectionId=""
time=2030-03-11T10:14:14.595Z level=INFO msg="Connected to websocket tunnel"
time=2030-03-11T10:14:14.596Z level=INFO msg="Starting server" "listening to"=wss://agent-01.humanitec.io/tunnel

Clean up

It’s best to store the public/private key pairs in a safe place for future use, e.g. in a secret store your organization is using. Refer to the product documentation of that store for more details on its use.

Then, remove the generated key pair from your local environment:

rm humanitec_agent_private_key.pem humanitec_agent_public_key.pem

Upgrade your installation

To upgrade your Humanitec Agent installation to the newest version, modify your installation to use the current image version number.

For Kubernetes, you can re-apply the above manifests . This will result in a new Agent Pod being created and the old one terminated.

Configuration

The following environment variables can be used to configure the Agent:

Variable Description
ORGS (Required) A comma-separated list of Humanitec Organizations that the Agent can be applied to.
PRIVATE_KEY_FILE (Optional) The absolute path of the private key. Default: /keys/private_key.pem
CONNECTION_ID (Optional) A string that can be used to identify different replicas of an Agent running under the same private key.
LOG_LEVEL (Optional) Specifies the log level of the Agent. One of DEBUG, INFO, WARN, or ERROR. Default: INFO.
AGENT_PROXY (Optional) Specifies an HTTP proxy to use to connect back to Humanitec via. It must be a valid URL and may optionally include username and password. The scheme must be either httpor https. (Available from version 1.8.0)
AGENT_PROXY_CA_CERT_FILE (Optional) If the scheme in AGENT_PROXY is https, this is the path to a file containing one or more Certificate Authority (CA) certificates as PEM encoded x509 certificates can be provided to use for connections to the proxy. NOTE: this will only be used for the proxy.
AGENT_PROXY_TLS_INSECURE_SKIP_VERIFY (Optional) If the scheme in AGENT_PROXY is https, setting this to true will mean that no attempt will be made to verify the certificate of the proxy. This is considered insecure.

Manage Humanitec Agents and keys

Get registered Humanitec Agents

To get currently registered Humanitec Agents for an Organization, use the following command:

humctl api get /orgs/${HUMANITEC_ORG}/agents

Get public keys for a Humanitec Agent

To get currently linked public keys for a Humanitec Agent in an Organization, use the following command:

humctl api get /orgs/${HUMANITEC_ORG}/agents/${AGENT_ID}/keys

De-register a Humanitec Agent

To de-register a Humanitec Agent from the Platform Orchestrator, use the following command. This will also remove all the public keys linked to the Agent:

humctl api delete /orgs/${HUMANITEC_ORG}/agents/${AGENT_ID}

First, create a pair of public/private key files as shown in these instructions .

To link the public key from the file humanitec_agent_public_key.pem to an existing Humanitec Agent, use the following command:

humctl api POST /orgs/${HUMANITEC_ORG}/agents/${AGENT_ID}/keys \
  -d '{
  "public_key": '"$(cat humanitec_agent_public_key.pem | jq -sR)"'
}'

Determine the fingerprint of the public key you wish to un-link by checking the list of registered public keys .

To unlink a particular public key from a Humanitec Agent, use the following command:

humctl api DELETE /orgs/${HUMANITEC_ORG}/agents/${AGENT_ID}/keys/<fingerprint>

Rotate the key pair for a Humanitec Agent

To rotate the key pair used by a registered Humanitec Agent instance, follow these steps:

  • Generate a new key pair as shown in these instructions .
  • Link the new public key to the registered Humanitec Agent as shown in these instructions .
  • Update the Agent configuration to use the new private key.
    • The exact steps depend on the execution environment the Agent instance is using.
    • For Kubernetes, this means updating the Secret holding the private key.
  • Restart the Agent to pick up the updated public key.
    • For Kubernetes, this means restarting the Agent Pod.
  • Un-link the old public key from the registered Humanitec Agent as shown in these instructions .

Uninstalling

To uninstall the Humanitec Agent from a Kubernetes cluster, execute the Kubernetes installation manifests , replacing kubectl apply with kubectl delete.

To uninstall the Humanitec Agent from a different execution environment, follow the instructions of that environment to uninstall a running container.

Troubleshooting

Troubleshoot connectivity

To check whether your Humanitec Agent installation has the required access to the Platform Orchestrator, you must verify that a secure connection can be made to the https://agent-01.humanitec.io API. The Humanitec Agent logs will also report errors if this connection cannot be made.

Kubernetes:

kubectl debug -it $(kubectl get pods -l app.kubernetes.io/name=humanitec-agent -n humanitec-agent -o name) \
  --image curlimages/curl -n humanitec-agent --arguments-only -- -I --fail https://agent-01.humanitec.io/tunnel

Docker:

# Run and attach to a debug container on the same Docker host
docker run -it curlimages/curl

# Execute in the container
curl -I --fail https://agent-01.humanitec.io/tunnel

The command will succeed with an HTTP 204 status if the connection succeeds.

Debugging network issues is beyond the scope of this document. Consider these aspects in case of connectivity issues:

  • Is there an egress route from the execution environment of the Humanitec Agent to the endpoint agent-01.humanitec.io?
  • Does egress traffic have to pass through a firewall or virtual appliance that inspects traffic and might block it on TCP port 443?
  • If the execution environment is attached to a virtual network/VPC, does that network provide flow logs that might provide insights?
  • For Kubernetes, are network policies affecting the egress of the Humanitec Agent pod?
  • Is an HTTP proxy required to reach the public internet?

HTTP response 401 in Agent logs

If you see this message in the Agent logs:

level=ERROR msg="Can't connect to websocket" err="unable to establish websocket connection: unexpected HTTP response status: 401, { []}"

It means the Agent could connect to the Platform Orchestrator on the network level, but was not authenticated.

Verify that the Agent’s public key is registered for all Organizations configured for the Agent.

Unable to determine cluster version: Bad Gateway

If a Deployment fails that utilizes a Humanitec Agent, and you see this error message in the Platform Orchestrator:

Unable to determine cluster version: Get "https://.35.197.242.20/version": Bad Gateway

It might be caused by one of the following issues:

  • The target Agent is not running for some reason.
  • The private key in private_key.pem is signing requests to the Platform Orchestrator with a key that has not been shared or validated yet.

Suggested mitigations are:

  • Check the status of the target Agent’s container and its logs.
  • Restart the Agent to re-establish the websocket tunnel with the Platform Orchestrator.
  • If the private key has been updated but not added to the Agent registered with the Platform Orchestrator, please do it as suggested here .

Next steps

Now that you’ve installed the Humanitec Agent, use it to deploy Workloads to a private Kubernetes cluster .

Overview

Helm Chart Values