Integrate

How to integrate your CI with your IDP #

Container Registries store and manage container images ready for use when they are needed in a deployment. Developers (or at least the CI pipelines they trigger) “push” built images into a registry and a Kubernetes cluster pull images out of a registry as needed - usually as part of a deployment.

There are many providers of container registries, and they can be provided as managed services or be self-hosted. The basic mechanics of pushing and pulling images to or from registries are the same across all providers; however, the way that these requests are authenticated can be very different across different providers.

Humanitec provides a way of centrally managing the credentials required to push or pull images out of a registry. This makes it easy to pull images from private registries even if they are hosted in a different cloud to where the Kubernetes cluster is hosted. For example, using Humanitec it is straightforward to pull images from a private Elastic Container Registry (ECR) in AWS from a Kubernetes cluster running in Azure. Humanitec ensures that the short-lived tokens required to authenticate with the ECR are generated and kept up to date in the Azure cluster.

Why does Humanitec need access to my registry? #

Humanitec actually does not need access to your container registry. The only parts of your infrastructure that need access are usually the CI pipeline to push the images into the registry and the Kubernetes cluster to pull the images during deployments. Humanitec can help out by being a central place to manage credentials that can be inserted into your CI pipeline or Kubernetes cluster.

In most cases, when everything is hosted on one cloud provider, everything works out of the box. That is, image pulls from a cluster will automatically be authenticated for Kubernetes clusters running in the same project or account.

In this case, it is not necessary to add your container registry credentials to Humanitec if you do not want Humanitec to provide credentials to the CI Pipeline.

Providing credentials to a CI Pipeline #

Pushing to registries such as ECR, Google Container Registry (GCR), or Google Artifact Registry (GAR) can be complex from 3rd party CI pipelines. This is because ECR, GCR, GAR do not support long-lived static credentials. It is necessary to fetch temporary credentials using either a command-line tool or directly by the API to authenticate with the registry.

If you have registered a registry with Humanitec, you can optionally provide credentials for the registry to your CI pipeline. This means your CI pipeline can fetch a set of valid credentials with a single API request using a Humanitec static token. The static token needs to be issued for a service user that has the ArtefactContributor role on the Humanitec Organization.

Here is an example request:

• CLI
• API

humctl api get /orgs/${HUMANITEC_ORG}/registries/${REGISTRY_ID}/creds

curl \
  -H "Authorization: Bearer ${HUMANITEC_TOKEN}" \
  -H "Content-Type: application/json" \
  "https://api.humanitec.io/orgs/${HUMANITEC_ORG}/registries/${REGISTRY_ID}/creds"

In the previous example:

• HUMANITEC_TOKEN contains the token issued for a service user which holds the ArtefactContributor on the organization.
• HUMANITEC_ORG is the organization.
• REGISTRY_ID is the ID chosen for the registry.

The response is a JSON object with username and password properties.

Creating registry credential secrets in Kubernetes #

Whenever a container registry is configured in Humanitec, any image that is hosted on that registry will cause a registry credential secret to being injected as part of the deployment.

The registry is determined by the longest prefix match of the supplied image path.

Adding a container registry to Humanitec #

The following example shows how to add a basic registry to Humanitec using the API.

Set the following environment variables for the CLI and API commands:

• Web UI
• CLI
• API
• Terraform

humctl api post /orgs/${HUMANITEC_ORG}/registries \
  -d '{
  "enable_ci": false,
  "id": "my-unique-id",
  "registry": "registry.example.com/my-team",
  "type": "basic",
  "creds": {
    "username": "'${REGISTRY_USERNAME}'",
    "password": "'${REGISTRY_PASSWORD}'"
  }  
}'

curl "https://api.humanitec.io/orgs/${HUMANITEC_ORG}/registries" \
  -H "Authorization: Bearer ${HUMANITEC_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
  "enable_ci": false,
  "id": "my-unique-id",
  "registry": "registry.example.com/my-team",
  "type": "basic",
  "creds": {
    "username": "'${REGISTRY_USERNAME}'",
    "password": "'${REGISTRY_PASSWORD}'"
    }
}'

Adding private Docker Hub registries #

Many systems treat Docker Hub registries as a special case and so will treat unqualified image paths as being hosted on Docker Hub by default. As Humanitec uses prefix matching to identify which registry a particular image path uses, it is important to set the “URL” field to include registry.hub.docker.com as a prefix to any private Docker Hub registry being added.

For example, if you have set up an organization in Docker Hub called my-organization then the URL should be registry.hub.docker.com/my-organization.

Build pipelines with your IDP #

CI pipelines inform your Internal Developer Platform whenever a new build is available and trigger automation rules. The following sections describes how you can connect your CI pipelines to your IDP.

The platform easily integrates with:

• Bitbucket
• CircleCI
• Codefresh
• GitHub
• GitLab
• Jenkins
• and many more pipelines

Container registries #

Container Registries securely store all of your organization’s container images. You can make use of Humanitec’s out-of-the-box image registry or connect your (existing) image registry with Humanitec. The platform easily integrates with JFrog Container Registry, Harbor, Elastic Container Registry, and many more registry types.

Integrate with your CI pipelines #

CI pipelines define the series of steps that must be performed to deliver a new version of software. Typical steps are build, test, validate / compliance, release, and deploy. The build step produces an image that is then pushed into a registry.

Humanitec as your Internal Developer Platform needs to know about all available builds. This allows users of Humanitec to use these builds in their Applications, and it’s the starting point for further automation.

In most setups, it’s sufficient to notify Humanitec at the end of your CI pipeline about a newly available build after it has been pushed to your registry. For more information, see Container Registries.

Humanitec also offers a hosted registry in case you do not have your own registry. Using the hosted registry requires an additional step in your CI pipeline to build the image and to push it to the Humanitec registry.

In most cases, the only change needed in existing CI pipelines to integrate with Humanitec is a curl or wget command to notify Humanitec about every new build available.

For more information, see Notify Humanitec.

Why does Humanitec offer a hosted registry? #

Almost all teams using Humanitec have their own registry either self-managed or managed by their cloud provider. Humanitec still offers a managed container registry that is compartmentalized for each organization in Humanitec. This registry is often used to test Humanitec since it is easy to just push an image with a shell script from a developer’s local machine to make it available within Humanitec without already changing existing CI pipelines. The required shell script can be found further down.

Manage image sources #

You need to add your CI pipelines as image sources to Humanitec. This allows Humanitec to pull images from your registry and make them available to your developers.

Add new image sources #

In the context of Humanitec, a CI pipeline is an image source. An image source provides Humanitec with container images which can then be deployed to the Environments of your Apps as Workloads.

Notify Humanitec #

New images can be registered with the Humanitec Platform Orchestrator through the artefact version endpoint. This notification can trigger an Artefact Automation Pipeline for automated deployment.

You can add one of the commands below to the end of your existing CI pipeline:

• CLI
• curl
• wget

humctl create artefact-version \
  --type container \
  --name "${IMAGE_NAME}"\
  --version "${IMAGE_TAG}" \
  --commit "${COMMIT_SHA}" \
  --ref "${GIT_REF}"

curl \
--request POST "https://api.humanitec.io/orgs/${HUMANITEC_ORG}/artefact-versions" \
--header "Authorization: Bearer ${HUMANITEC_TOKEN}" \
--header "Content-Type: application/json" \
--data-raw '{
    "name": "'${IMAGE_NAME}'",
    "version": "'${IMAGE_TAG}'",
    "type": "container",
    "commit": "'${COMMIT_SHA}'",
    "ref": "'${GIT_REF}'"
}'

wget --quiet \
 --method POST \
 --timeout=0 \
 --header "Authorization: Bearer ${HUMANITEC_TOKEN}" \
 --header "Content-Type: application/json" \
 --body-data '{
    "name": "'${IMAGE_NAME}'",
    "version": "'${IMAGE_TAG}'",
    "type": "container",
    "commit": "'${COMMIT_SHA}'",
    "ref": "'${GIT_REF}'"
}' \
"https://api.humanitec.io/orgs/${HUMANITEC_ORG}/artefact-versions"

Placeholder variables in the example above:

• HUMANITEC_TOKEN contains a Humanitec Static Token.
• IMAGE_NAME represents the full image name excluding the tag. It should include the registry and repository. For example: registry.example.com/project/my-image.
• IMAGE_TAG is the tag for the image.
• COMMIT_SHA is the git SHA that the image was built from.
• GIT_REF is the git reference that the image was built from. For example: refs/heads/main, refs/tags/1.3.4 or refs/pull/39/merge.

You can integrate Humanitec in any CI pipeline by adding the required steps yourself. You can find example bash scripts in Write your own integration.

GitHub Actions workflow #

Humanitec offers an out-of-the-box CI pipeline integration for GitHub Actions. The source code of the GitHub Action is available in this public GitHub repository: Build and Push to Humanitec.

If you do not have a GitHub Actions workflow set up, you will need to create one:

1. Go to the GitHub repository you want to connect.
2. Select the Actions tab.
3. Select Set up a workflow yourself in the top right corner.
4. Remove all the lines after the line that says: - uses: actions/checkout@v2. The resulting file should look like this:

# This is a basic workflow to help you get started with Actions.
name: CI

# Controls when the action will run. Triggers the workflow on push or pull request
# events but only for the main branch
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
      - uses: actions/checkout@v2

Once the GitHub Actions workflow is created, you can continue to define the action that notifies Humanitec when a new image version is available. For safety please create the secret first and the code change second as described below:

1. Add the HUMANITEC_TOKEN as a repository secret:
   1. In the repository you want to connect, go to Settings.
   2. Select Secrets > New repository secret and create a new secret called HUMANITEC_TOKEN and enter the token.
2. Append the Build and Push to Humanitec code snippet to the end of the YAML file.

Add new Workload artefacts #

“Workload” artefacts contain a Score Workload specification which includes all the details required to launch your service.

Notify Humanitec #

Whenever a new version of your Workload is ready, you can notify the Humanitec Platform Orchestrator by uploading the Score Workload specification to the artefact version endpoint. This notification can trigger an Artefact Automation Pipeline for automated deployment.

You can add one of the commands below to the end of your existing CI pipeline.

Set the following environment variables:

• CLI
• API

humctl create artefact-version \
  --type workload \
  --spec ./score.yaml \
  --name myProject/myWorkloadArtefact \
  --version 1.2.3 \
  --ref refs/heads/main

curl https://api.humanitec.io/orgs/${HUMANITEC_ORG}/artefact-versions \
  -H "Authorization: Bearer ${HUMANITEC_TOKEN}" \
  -F type=workload \
  -F name=myProject/myWorkloadArtefact \
  -F [email protected] \
  -F ref=refs/heads/main

Write your own integration #

You can integrate Humanitec with any CI pipeline or even use Humanitec from a terminal on your local machine in case you just want to give it a test without changing your existing CI pipelines. This section provides all required information and bash scripts to do that.

Full bash script for local usage #

If you want to test Humanitec from your local machine (by pushing a local image to the registry provided by Humanitec), You can use the following shell script:

#!/bin/sh

print_usage ()
{
 cat <<EOF
USAGE:
​
  ./local_build_push_notify.sh [options] IMAGE_NAME [TAG]
​
OPTIONS:
​
  -t, --token
      The token to pass into the authorization header. Will override the
      HUMANITEC_TOKEN environment variable if supplied.
​
  -o, --org
      The organization in Humanitec that the token belongs to.
​
  -h, --help
      Show this help text.
​
NOTES:
​
  IMAGE_NAME represents the full image name excluding the tag. It should include
  the registry and the repository. For example "registry.example.com/project/my-image".
​
  TAG is the tag for the image. If TAG is not provided, it will be set to the current commit SHA1.
​
  By default, the token will be read from the HUMANITEC_TOKEN environment
  variable.
​
EXAMPLE:
​
  ./local_build_push_notify.sh --org my-org registry.example.com/project/my-image 0.3.2-rc5
​
EOF
}

key_from_json_obj ()
{
 tr -d '\n' | sed 's/^[ \t\v\f]*{.*"'"${1}"'"[ \t\v\f]*:[ \t\v\f]*"\([^"]*\)"[ \t\v\f]*[,}].*$/\1/'
}

fetch_url ()
{
 method="$1"
 payload=""
 if [ "$method" = "POST" ] || [ "$method" = "PUT" ] || [ "$method" = "PATCH" ]
 then
  payload="$2"
  shift
 fi
 url="$2"
 auth_header="Authorization: Bearer ${HUMANITEC_TOKEN}"
 if command -v curl &> /dev/null
 then
  if [ "$payload" != "" ]
  then
   curl --fail -s \
    -X "$method" \
    -H "$auth_header" \
    -H "Content-Type: application/json" \
    -d "$payload" \
    "$url"
  else
   curl --fail -s \
    -X "$method" \
    -H "$auth_header" \
    "$url"
  fi
     elif command -v wget &> /dev/null
 then
  if [ "$payload" != "" ]
  then
   wget --quiet -O - \
    --method="$method" \
    --header="$auth_header" \
    --header="Content-Type: application/json" \
    --body-data="$payload" \
    "$url"
  else
   wget --quiet -O - \
    --method="$method" \
    --header="$auth_header" \
    "$url"
  fi
 else
  echo "System does not have the commands wget or curl installed." >&2
  exit 1
 fi
}

api_prefix="https://api.humanitec.io"

while (( $# ))
do
 case "$1" in
  '-t'|'--token')
   export HUMANITEC_TOKEN="$2"
   shift
   ;;
  '-o'|'--org')
   export HUMANITEC_ORG="$2"
   shift
   ;;
  '--api-prefix')
   api_prefix="$2"
   shift
   ;;
  '-h'|'--help')
   print_usage
   exit
   ;;
  *)
   image_name="$1"
   if [[ $2 == ""  ||  $2 == -* ]]
   then
    image_tag=""
    image_with_tag="${image_name}"
   else
    image_tag="$2"
    image_with_tag="${image_name}:${image_tag}"
    shift
   fi
 esac
 shift
done

if [ -z "${HUMANITEC_TOKEN}" ]
then
 echo "No token specified as option or via HUMANITEC_TOKEN environment variable." >&2
 exit 1
fi

if [ -z "$HUMANITEC_ORG" ]
then
 echo "No Organization specified as option or via HUMANITEC_ORG environment variable." >&2
 exit 1

fi

if [ -z "$image_with_tag" ]
then
 echo "No IMAGE_NAME provided." >&2
 exit 1
fi

echo "Retrieving registry credentials"
registry_json="$(fetch_url GET "${api_prefix}/orgs/${HUMANITEC_ORG}/registries/humanitec/creds")"
if [ $? -ne 0 ]
then
 echo "Unable to retrieve credentials for humanitec registry." >&2
 exit 1
fi

username="$(echo "$registry_json" | key_from_json_obj "username")"
password="$(echo "$registry_json" | key_from_json_obj "password")"
server="$(echo "$registry_json" | key_from_json_obj "registry")"

ref="$(git rev-parse --symbolic-full-name HEAD)"
commit="$(git rev-parse HEAD)"

if [ "$image_tag" = "" ]
then
 image_tag="$commit"
fi

echo "Logging into docker registry"
echo "${password}" | docker login -u "${username}" --password-stdin "${server}"
if [ $? -ne 0 ]
then
 echo "Unable to log into humanitec registry." >&2
 exit 1
fi

echo "Performing docker build"
local_tag="${HUMANITEC_ORG}/${image_name}:${image_tag}"
if ! docker build -t "$local_tag" .
then
 echo "Docker build failed." >&2
 exit 1
fi

remote_tag="${server}/$local_tag"
if ! docker tag "$local_tag" "$remote_tag"
then
 echo "Error pushing to remote registry: Cannot retag locally." >&2
 exit 1
fi

echo "Pushing image to registry: $remote_tag"
if ! docker push "$remote_tag"
then
 echo "Error pushing to remote registry: Push failed." >&2
 exit 1
fi

echo "Notifying Humanitec"
payload="{\"commit\":\"${commit}\",\"ref\":\"${ref}\",\"version\":\"${image_tag}\",\"name\":\"${image_name}\",\"type\":\"container\"}"
if ! fetch_url POST "$payload" "${api_prefix}/orgs/${HUMANITEC_ORG}/artefact-versions"
then
        echo "Unable to notify Humanitec." >&2
        exit 1
fi