Deployment Pipelines

Notify slack channel of failed deployments

Humanitec Pipelines supports an http action which is able to send external requests with data from the Pipeline execution. This can be used to trigger or notify external systems. One example of this is to notify a channel in Slack or raise a support request or ticket.

In the following example, we’ll notify a Slack channel of the success or failure of all deployments through the Pipeline.

The Slack credential is stored as a secret Application Shared Value .

pipeline.yaml ( view on GitHub ) :

name: Deployment with Notifications
on:
  deployment_request: {}

# Inherit the permissions of the user triggering the deployment
permissions:
  run-as: inherit

jobs:
  deploy:
    # This critical property ensures that we don't stop the pipeline when the initial deployment fails.
    continue-on-error: true
    steps:
    - if: ${{ ! inputs.set_id }}
      name: Create Deployment Set
      id: create-deployment-set
      uses: actions/humanitec/apply@v1
      with:
        delta_id: ${{ inputs.delta_id }}
        env_id: ${{ inputs.env_id }}

    - name: Deploy Set To Environment
      uses: actions/humanitec/deploy@v1
      with:
        set_id: ${{ inputs.set_id || steps.create-deployment-set.outputs.set_id }}
        value_set_version_id: ${{ inputs.value_set_version_id }}
        env_id: ${{ inputs.env_id }}
        message: ${{ inputs.comment }}

  notify:
    needs: ["deploy"]
    steps:
    - name: Notify Slack
      uses: actions/humanitec/http@v1
      with: 
        method: POST
        url: ${{ app.values.SLACK_WEBHOOK }}
        headers:
          "Content-Type": "application/json"
        data:
          text: |
            Deployment to ${{ inputs environment }} ${{ needs.deploy.status }}.
            Comment: ${{ inputs.comment }}
            Link: "https://app.humanitec.io/orgs/${{ pipeline.org.id }}/apps/${{ pipeline.app.id }}/envs/${{ inputs.environment }}"

    # Now fail the pipeline if the initial deployment failed
    - name: Fail if failed
      if: ${{ needs.deploy.status == 'failed' }}
      uses: actions/humanitec/fail@v1
      with:
        message: Deployment failed.

Prevent direct deployment to production

In the “promote between environments” example, a Pipeline is used to promote a Workload from development to production. To ensure that all changes to production have first been tested in development, it may be necessary to block direct deployment of Deltas to production. Deployment request Matching Criteria can be used for this.

First, create a Pipeline that always fails with a log message.

Then link this Pipeline to the production Environment with the following Matching Criteria :

  • app_id: my-application
  • env_id: production
  • deployment_type: deploy

The deployment_type: deploy part effectively excludes the re-deploy situation because you still want to be able to rollback production to a previous release without re-promoting a change through the promotion pipeline. This can be relevant e.g. during an incident or other operational needs.

Because this Pipeline is triggered by deployment_request, any direct deployments through the user interface and APIs will run through this Pipeline and fail before the deployment takes place.

Deployments through other Pipelines, such as the aforementioned promotion Pipeline, will still continue as normal.

pipeline.yaml ( view on GitHub ) :

name: No deployment allowed
on:
  deployment_request: {}
permissions:
  application: viewer
jobs:
  log:
    steps:
    - uses: actions/humanitec/fail@v1
      with:
        message: Direct deployment to ${{ inputs.env_id }} is not allowed.

Run automated tests after deployment

When the Platform Orchestrator deploys changes into an Environment, the Deployment status indicates whether it successfully provisioned associated resources and configured the Workloads. It does not necessarily imply that the Applications are healthy or that the desired functionality is available to end users. This is why a major benefit of Continuous Delivery Pipelines is the ability to perform health checks and run a suite of tests against the running application in a pre-production or even production Environment.

Humanitec Pipelines can be used to achieve this goal by combining the common deployment steps with steps that wait for readiness, and execute external tests in systems such as GitHub Actions or GitLab Pipelines.

The example Pipeline uses a GitHub Action.

Once configured for the target environment using deployment request Matching Criteria, this Pipeline will now fail automatically when any of the following happens:

  1. The deployment fails, due to reasons like invalid configuration, inaccessible Kubernetes clusters, or failures to provision resources in the Platform Orchestrator
  2. Workloads that fail to reach a ready state
  3. Failures occur in the attached GitHub Actions Workflow

All of these events are valuable health tests that can be automated by Humanitec Pipelines.

pipeline.yaml ( view on GitHub ) :

name: Testing after deployment

# This pipeline is a deployment request pipeline and runs for any deploy or re-deploy.
on:
  deployment_request: {}

# This pipeline needs permissions to deploy to our development environments. We could add staging and production here too.
permissions:
  application: developer
  env-types:
    development: deployer

# Because this pipeline runs tests, we're going to use concurrency controls to ensure only one deploy and test cycle is running at a time against the target environment.
concurrency:
  group: "${{ pipeline.id }}-${{ pipeline.app.id }}-${{ inputs.env_id }}"

jobs:
  deploy:
    steps:
    - if: ${{ ! inputs.set_id }}
      name: Create Deployment Set
      id: create-deployment-set
      uses: actions/humanitec/apply@v1
      with:
        delta_id: ${{ inputs.delta_id }}
        env_id: ${{ inputs.env_id }}

    - name: Deploy Set To Environment
      uses: actions/humanitec/deploy@v1
      with:
        set_id: ${{ inputs.set_id || steps.create-deployment-set.outputs.set_id }}
        value_set_version_id: ${{ inputs.value_set_version_id }}
        env_id: ${{ inputs.env_id }}
        message: ${{ inputs.comment }}

    - name: Wait For Ready
      uses: actions/humanitec/wait-for-readiness@v1
      with:
        env_id: ${{ inputs.env_id }}
 
    - name: Run Tests
      uses: actions/humanitec/github-workflow@v1
      with:
        repo: my-github-organization/my-github-repository
        ref: main
        workflow: ".github/workflows/push.yml"
        # Read access token from a secret Application value
        access_token: "${{ app.values.github-token }}"
        # Pass inputs to the pipeline that direct the tests to the target environment
        inputs:
          environment: ${{ inputs.env_id }}
        request_uid_input: request_uid

Wait for workloads readiness or fail

When the Platform Orchestrator deploys changes into an Environment, the Deployment status indicates whether it successfully provisioned associated resources and configured the Workloads. It does not necessarily imply that the Workloads are healthy or that the desired functionality is available to end users. The Pipeline will also be successful in scenario where the Workload will never be Ready, for example when the container image doesn’t exist (ImagePullBackOff), or when an admission controller is blocking the deployment in the Kubernetes cluster, etc.

Humanitec Pipelines can be used to wait for the readiness of the Worklaods deployed in Kubernetes, and fail the Pipeline if the previous step timed out.

This Pipeline definition shows how to accomplish this.

Note: the Deployment status on the Deployment in the Platform Orchestrator will still be Successful.

When the Pipeline is in Failed status, humctl score deploy --wait or humctl deploy --wait will fail. Then to get the associated errors, you can catch the details of errors by combining the result of:

  • humctl get deployment-error
  • humctl api get /orgs/${HUMANITEC_ORG}/apps/${APP}/envs/${ENV}/runtime

pipeline.yaml ( view on GitHub ) :

name: Wait for Workloads readiness or fail

# This pipeline is a deployment request pipeline and runs for any deploy or re-deploy.
on:
  deployment_request: {}

# This pipeline needs permissions to deploy to our development environments. We could add staging and production here too.
permissions:
  application: developer
  env-types:
    development: deployer

# Because this pipeline runs tests, we're going to use concurrency controls to ensure only one deploy and test cycle is running at a time against the target environment.
concurrency:
  group: "${{ pipeline.id }}-${{ pipeline.app.id }}-${{ inputs.env_id }}"

jobs:
  deploy:
    steps:
    # Taken from default Pipeline definition.
    # If there is no existing deployment set in the request, create one by applying the delta to the target environment.
    - if: ${{ ! inputs.set_id }}
      name: Create Deployment Set
      id: create-deployment-set
      uses: actions/humanitec/apply@v1
      with:
        delta_id: ${{ inputs.delta_id }}
        env_id: ${{ inputs.env_id }}

    # Taken from default Pipeline definition.
    # Finally deploy the set to the target environment with a custom message and optional value set. 
    - name: Deploy Set To Environment
      uses: actions/humanitec/deploy@v1
      with:
        set_id: ${{ inputs.set_id || steps.create-deployment-set.outputs.set_id }}
        value_set_version_id: ${{ inputs.value_set_version_id }}
        env_id: ${{ inputs.env_id }}
        message: ${{ inputs.comment }}

    # Waits until all Workloads in an Environment are in a “ready” state, otherwise fails the Pipeline Step after a timeout.
    - name: Wait for Workloads readiness
      continue-on-error: true # Optional, if you want to have another step after this one in case of failure to write a custom message, send a message in Slack, etc.
      id: wait-for-workloads-readiness
      uses: actions/humanitec/wait-for-readiness@v1
      with:
        env_id: ${{ inputs.env_id }}
        timeout_minutes: 2

    # (Optional) Fails the Pipeline Step and writes a custom error message. It's an example that you can adapt and replace by other actions like send a message in Slack, etc.
    - if: ${{ steps.wait-for-workloads-readiness.status == 'failed' }}
      name: Fail if Workloads readiness failed
      uses: actions/humanitec/fail@v1
      with:
        message: Workloads readiness failed, check the Runtime errors.

Artefact Automation Pipelines