serverless-ecs

The serverless-ecs runner type lets you execute runners on AWS ECS (Elastic Container Service) Fargate clusters within your AWS account.

The Platform Orchestrator can execute the runner as an ECS task for each deployment. This task will run on your ECS cluster which means:

It stores its state in your choice of state storage backend within your AWS account
It is bound to a subnet within your own VPC for networking
It executes as an IAM role within your account and may read and write infrastructure within your account according to the permissions you delegate to it

To access the ECS cluster, the Platform Orchestrator uses temporary credentials granted through OIDC and an assume-role-with-web-identity policy to assume a role within your AWS account.

Although you can use most state storage backends with the ECS runner, we recommend you use the s3 state storage type for the best integration and centralised management.

Example Configuration

resource.tf (view on GitHub ) :

resource "platform-orchestrator_serverless_ecs_runner" "example" {
  id          = "my-ecs-runner"
  description = "This is a sample ECS runner configuration."

  runner_configuration = {
    auth = {
      role_arn = "arn:aws:iam::123456789012:role/humanitec_role"
    }
    job = {
      region             = "eu-central-1"
      cluster            = "my-ecs-cluster-name"
      execution_role_arn = "arn:aws:iam::123456789012:role/execution_role"
      subnets            = ["my-subnet-1"]

      task_role_arn        = "arn:aws:iam::123456789012:role/task_role"
      is_public_ip_enabled = false
      security_groups      = []

      environment = {
        "EXAMPLE_ENVIRONMENT_VARIABLE" = "value"
      }

      secrets = {
        "SECRET_ENVIRONMENT_VARIABLE"   = "arn:aws:secretsmanager:eu-central-1:123456789012:secret:myapp/api-key-XyZ9Qw"
        "PROPERTY_ENVIRONMENT_VARIABLE" = "arn:aws:ssm:eu-central-1:123456789012:parameter/app/config/api-endpoint"
      }
    }
  }

  state_storage_configuration = {
    type = "s3"
    s3_configuration = {
      bucket      = "humanitec-ecs-runner-state"
      path_prefix = "state-files"
    }
  }
}

runner-config.yaml:

runner_configuration:
  type: serverless-ecs
  auth:
    # The role the Platform Orchestrator assumes within your account
    role_arn: arn:aws:iam::123456789012:role/orchestrator-role
  job:
    region: eu-central-1
    cluster: my-ecs-cluster
    # The role the task executes with
    execution_role_arn: arn:aws:iam::123456789012:role/runner-execution-role
    subnets: ["subnet-a"]
state_storage_configuration:
  ...

Configuration options

The following configuration shows all available options for the serverless-ecs runner type:

Refer to the resource schema in the Terraform or OpenTofu provider documentation.

runner_configuration:
  # Runner type
  type: serverless-ecs

  auth:
    # The ARN of the role that the Platform Orchestrator will assume in your account.
    role_arn: arn:aws:iam::123456789012:role/orchestrator-role

  job:
    # The AWS region of the ECS cluster
    region: us-west-2

    # The name of the AWS ECS cluster to execute tasks in
    cluster: my-ecs-cluster

    # The ARN of the role that will be used to execute the task. You should grant this role at least the `arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy` or equivalent permissions. This role will also be used inside the task if no `task_role_arn` is defined.
    execution_role_arn: arn:aws:iam::123456789012:role/runner-execution-role

    # A list of at least one AWS subnet ID within your account to use as the networking for the task.
    subnets:
    - subnet-1

    # An optional ARN of the role that will be used inside the task. This defaults to the `execution_role_arn` if you don't define it. You should grant this role at least the permissions to use the target state store configuration.
    task_role_arn: arn:aws:iam::123456789012:role/runner-task-role

    # An optional boolean to request a public IP for each task. This may be required in subnets using an internet gateway for egress routing.
    is_public_ip_enabled: true

    # An optional list of security groups to associate with the ECS tasks.
    security_groups:
    - runner-security-group

    # An optional map of plain-text environment variables to populate in the runner.
    environment:
      EXAMPLE_ENVIRONMENT_VARIABLE: value

    # An optional of environment variable names with either AWS Secrets Manager or AWS Parameter Store parameters to mount in the runner.
    secrets:
      SECRET_ENVIRONMENT_VARIABLE: "arn:aws:secretsmanager:eu-central-1:123456789012:secret:myapp/api-key-XyZ9Qw"
      PROPERTY_ENVIRONMENT_VARIABLE: "arn:aws:ssm:eu-central-1:123456789012:parameter/app/config/api-endpoint"

state_storage_configuration:
  # ...

The environment property is for plain text and non sensitive values only since it is stored in the database of the Platform Orchestrator. All sensitive values should be mounted from secrets.

Setting sensitive environment variables

ECS runners support a runner_configuration.job.secrets field which may contain a mapping from environment variable key to AWS Secrets Manager Secret ARN or AWS Systems Manager Parameter ARN:

  runner_configuration = {
    job = {
      secrets = {
        TF_EXAMPLE   = "arn:aws:secretsmanager:eu-central-1:123456789012:secret:myapp/api-key-XyZ9Qw"
        TF_EXAMPLE_2 = "arn:aws:ssm:eu-central-1:123456789012:parameter/app/config/api-endpoint"
      }
    }
  }

runner_configuration:
  job:
    secrets:
      TF_EXAMPLE: arn:aws:secretsmanager:eu-central-1:123456789012:secret:myapp/api-key-XyZ9Qw
      TF_EXAMPLE_2: arn:aws:ssm:eu-central-1:123456789012:parameter/app/config/api-endpoint

We recommend using secrets manager ARNs for sensitive values.

The job.execution_role_arn of the ECS runner will be used to access these ARNs, therefor you will need to create an appropriate IAM policy to grant the required actions:

[
    {
        "Version": "2012-10-17",
        "Action": [
            "secretsmanager:GetSecretValue"
        ],
        "Resource": "arn:aws:secretsmanager:eu-central-1:123456789012:secret:myapp/*"
    },
    {
        "Version": "2012-10-17",
        "Action": [
            "ssm:GetParameter"
        ],
        "Resource": "arn:aws:ssm:eu-central-1:123456789012:parameter/app/config/api-endpoint"
    }
]

Required permissions

`auth.role_arn`

The Platform Orchestrator will assume this role using OIDC and use it to manage ECS task definitions and execute one-off tasks. Therefore it needs:

OIDC federation to allow the oidc.humanitec.dev provider to assume the role where the sub (subject) claim is ${HUMANITEC_ORG_ID}+${RUNNER_ID}
A policy allowing it to be passed to the ecs-tasks service
Permissions to List, Register, Deregister, and Delete ECS task definitions in the account
Permissions to List, Describe, Run, and Tag ECS tasks within the target ECS cluster

`job.execution_role_arn`

The Platform Orchestrator will use this role to setup and launch each ECS task for the runner. Therefore it needs:

A policy allowing it to be passed to the ecs-tasks service

No additional permissions are required. However, because this role will be used as the job.task_role_arn if you do not specify one, it may need additional runtime permissions. See the next section.

If your ECS cluster has Container Insights, KMS encryption, or other configuration settings that require additional permissions, you will need to assign them to this role.

`job.task_role_arn`

The Platform Orchestrator will expose this role within each ECS task for the runner using the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI and the local ECS container endpoint (http://169.254.179.2). This allows any AWS SDK within the runner to assume this role. This will be used by default for the s3 state storage backend or hashicorp/aws provider if used.

At minimum this role needs a policy allowing it to be assumed by the ecs-tasks service.

Setup guide

The steps below serve as a general guide of the steps to configure the runner using the either Terraform/OpenTofu (“TF”) code or the CLI. Your unique use case may require a more complex setup.

Before you begin

An existing AWS ECS cluster with Fargate serverless capacity enabled
An existing AWS Subnet ID within the same region as the ECS cluster. The subnet must have egress internet access configured via an appropriate gateway setup
The aws CLI installed and authenticated with a principal having permission to manage IAM roles and policies in the AWS account of the ECS cluster
Depending on your choice of tooling:
- TF: Either the Terraform CLI or the OpenTofu CLI installed
- CLI: The hctl CLI installed and authenticated against your Orchestrator organization

Prepare the cloud environment

Perform the following to allow the runner to execute:

Set values

locals {
  aws_account_id = "<my-aws-account-id>"
  aws_region     = "<my-aws-region>"
  ecs_cluster    = "<my-ecs-cluster-name>"
  subnet_id      = "<my-subnet-id>"
  humanitec_org  = "<my-org>"
}

export AWS_ACCOUNT_ID=<my-aws-account-id>
export AWS_REGION=<my-aws-region>
export ECS_CLUSTER=<my-ecs-cluster-name>
export SUBNET_ID=<my-subnet-id>
export HUMANITEC_ORG=<my-org>

Create an IAM role for the Orchestrator, creating the open-id-connect provider if you don’t have one already

locals {
  ecs_runner_id = "ecs-${local.ecs_cluster}-${local.aws_region}"
}

resource "aws_iam_openid_connect_provider" "provider" {
  url = "https://oidc.humanitec.dev"
  client_id_list = [
    "sts.amazonaws.com",
  ]
}

data "aws_iam_policy_document" "ecs_runner_humanitec_role" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]

    principals {
      type        = "Federated"
      identifiers = [aws_iam_openid_connect_provider.provider.arn]
    }

    condition {
      test     = "StringEquals"
      variable = "oidc.humanitec.dev:aud"
      values   = ["sts.amazonaws.com"]
    }

    condition {
      test     = "StringEquals"
      variable = "oidc.humanitec.dev:sub"
      values   = ["${local.humanitec_org}+${local.ecs_runner_id}"]
    }
  }
}

resource "aws_iam_role" "ecs_runner_humanitec_role" {
  name_prefix        = "ecs_humanitec"
  assume_role_policy = data.aws_iam_policy_document.ecs_runner_humanitec_role.json
  description        = "Role for Humanitec Orchestrator to access ECS clusters for launching runners"
}

aws iam create-open-id-connect-provider \
  --url https://oidc.humanitec.dev \
  --client-id-list sts.amazonaws.com

export RUNNER_ID=ecs-${ECS_CLUSTER}-${AWS_REGION}

cat <<EOF > humanitec-runner-trust-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/oidc.humanitec.dev"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.humanitec.dev:aud": "sts.amazonaws.com",
          "oidc.humanitec.dev:sub": "${HUMANITEC_ORG}+${RUNNER_ID}"
        }
      }
    }
  ]
}
EOF

export HUMANITEC_IAM_ROLE=ecs-runner-humanitec-role

aws iam create-role \
  --role-name ${HUMANITEC_IAM_ROLE} \
  --assume-role-policy-document file://humanitec-runner-trust-policy.json \
  --description "Role for Humanitec Orchestrator to access ECS clusters for launching runners"

Create an IAM policy with minimal ECS permissions and assign it to the role

resource "aws_iam_role_policy" "ecs_runner_humanitec_role" {
  name_prefix = "ecs_humanitec"
  role = aws_iam_role.ecs_runner_humanitec_role.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
        {
            Effect = "Allow",
            Action = [
                "ecs:ListTaskDefinitions",
                "ecs:DeregisterTaskDefinition"
            ],
            Resource = "*"
        },
        {
            Effect = "Allow",
            Action = [
                "ecs:RegisterTaskDefinition",
                "ecs:DeleteTaskDefinitions"
            ],
            Resource = [
              "arn:aws:ecs:${local.aws_region}:${local.aws_account_id}:task-definition/humanitec_*",
            ]
        },
        {
            Effect = "Allow",
            Action = [
                "ecs:ListTasks",
                "ecs:DescribeTasks",
                "ecs:RunTask",
                "ecs:TagResource",
            ],
            Resource = [
              "arn:aws:ecs:${local.aws_region}:${local.aws_account_id}:task-definition/humanitec_*",
              "arn:aws:ecs:${local.aws_region}:${local.aws_account_id}:cluster/${local.ecs_cluster}",
              "arn:aws:ecs:${local.aws_region}:${local.aws_account_id}:task/${local.ecs_cluster}/*",
              "arn:aws:ecs:${local.aws_region}:${local.aws_account_id}:container-instance/${local.ecs_cluster}/*"
            ]
        },
        {
            Effect = "Allow",
            Action = "iam:PassRole",
            Resource = "*",
            Condition = {
                StringLike = {
                    "iam:PassedToService" = "ecs-tasks.amazonaws.com"
                }
            }
        },
    ]
  })
}

cat <<EOF > humanitec-role-policy.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecs:ListTaskDefinitions",
        "ecs:DeregisterTaskDefinition"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ecs:RegisterTaskDefinition",
        "ecs:DeleteTaskDefinitions"
      ],
      "Resource": [
        "arn:aws:ecs:${AWS_REGION}:${AWS_ACCOUNT_ID}:task-definition/humanitec_*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ecs:ListTasks",
        "ecs:DescribeTasks",
        "ecs:RunTask",
        "ecs:TagResource"
      ],
      "Resource": [
        "arn:aws:ecs:${AWS_REGION}:${AWS_ACCOUNT_ID}:task-definition/humanitec_*",
        "arn:aws:ecs:${AWS_REGION}:${AWS_ACCOUNT_ID}:cluster/${ECS_CLUSTER}",
        "arn:aws:ecs:${AWS_REGION}:${AWS_ACCOUNT_ID}:task/${ECS_CLUSTER}/*",
        "arn:aws:ecs:${AWS_REGION}:${AWS_ACCOUNT_ID}:container-instance/${ECS_CLUSTER}/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "*",
      "Condition": {
        "StringLike": {
          "iam:PassedToService": "ecs-tasks.amazonaws.com"
        }
      }
    }
  ]
}
EOF

aws iam put-role-policy \
  --role-name ${HUMANITEC_IAM_ROLE} \
  --policy-name ecs-runner-humanitec-role-policy \
  --policy-document file://humanitec-role-policy.json

Create an IAM role to use as the execution role

data "aws_iam_policy_document" "ecs_runner_exec_role" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["ecs-tasks.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "ecs_runner_exec_role" {
  name_prefix = "ecs_exec"
  assume_role_policy = data.aws_iam_policy_document.ecs_runner_exec_role.json
}

cat <<EOF > ecs-runner-exec-role-trust.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ecs-tasks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

export EXEC_IAM_ROLE=ecs-runner-exec-role

aws iam create-role \
  --role-name ${EXEC_IAM_ROLE} \
  --assume-role-policy-document file://ecs-runner-exec-role-trust.json \
  --description "Role for Humanitec Orchestrator to launch ECS tasks"

You can repeat the previous steps to create the task role. Note that this is the role that will be available from within the ECS runner, so if you need to use an S3 state storage configuration or provision AWS infrastructure you should attach an appropriate policy to this role.

data "aws_iam_policy_document" "ecs_runner_task_role" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["ecs-tasks.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "ecs_runner_task_role" {
  name_prefix = "ecs_task"
  assume_role_policy = data.aws_iam_policy_document.ecs_runner_task_role.json
}

cat <<EOF > ecs-runner-task-role-trust.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ecs-tasks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

export TASK_IAM_ROLE=ecs-runner-task-role

aws iam create-role \
  --role-name ${TASK_IAM_ROLE} \
  --assume-role-policy-document file://ecs-runner-task-role-trust.json \
  --description "Role for Humanitec Orchestrator to use within ECS tasks"

Or re-use the execution role here:

export TASK_IAM_ROLE=${EXEC_IAM_ROLE}

Configure a state storage

Decide which state storage types the runner is supposed to use. Check the compatibility matrix and perform the setup according to the chosen state storage type documentation.

Prepare a TF code snippet containing the properties for the state storage configuration underneath a property state_storage_configuration:

  state_storage_configuration = {
    ...
  }

Prepare a local file state-storage-config.yaml containing the properties for the state storage configuration underneath the top level property state_storage_configuration.

state_storage_configuration:
  ...

Create a runner

Prepare the runner configuration

resource "platform-orchestrator_serverless_ecs_runner" "runner" {
  id = local.ecs_runner_id
  runner_configuration = {
    auth = {
      role_arn = aws_iam_role.ecs_runner_humanitec_role.arn
    }
    job = {
      region = local.aws_region
      cluster = local.ecs_cluster
      execution_role_arn = aws_iam_role.ecs_runner_exec_role.arn
      task_role_arn = aws_iam_role.ecs_runner_task_role.arn
      subnets = [local.subnet_id]
    }
  }
  state_storage_configuration = {
    # ...
  }
}

Prepare the runner_configuration in a local file:

cat <<EOF > runner-config.yaml
runner_configuration:
  type: serverless-ecs
  auth:
    role_arn: arn:aws:iam::${AWS_ACCOUNT_ID}:role/${HUMANITEC_IAM_ROLE}
  job:
    region: ${AWS_REGION}
    cluster: ${ECS_CLUSTER}
    subnets: [${SUBNET_ID}]
    execution_role_arn: arn:aws:iam::${AWS_ACCOUNT_ID}:role/${EXEC_IAM_ROLE}
    task_role_arn: arn:aws:iam::${AWS_ACCOUNT_ID}:role/${TASK_IAM_ROLE}
EOF

Append the state storage configuration you created earlier

Append the state storage config file prepared in the previous section to the existing runner configuration:

Add the state_storage_configuration block you prepared to the platform-orchestrator_serverless_ecs_runner resource:

resource "platform-orchestrator_serverless_ecs_runner" "example" {
  
  # ...

  state_storage_configuration = {
    # Your prepared state storage configuration
    # ...
  }
}

cat state-storage-config.yaml >> runner-config.yaml

Verify the runner configuration

Verify the structure of the runner configuration. It needs to have the top level properties as shown:

resource "platform-orchestrator_serverless_ecs_runner" "example" {
  
  runner_configuration = {
    # ...
  }
  state_storage_configuration = {
    # ...
  }
}

Verify the structure of the configuration file. It needs to have the top level properties as shown:

cat runner-config.yaml

runner_configuration:
  ...
state_storage_configuration:
  ...

Create the runner

Create the runner using the configuration prepared previously:

apply the TF configuration you created.

hctl create runner ${RUNNER_ID} \
  [email protected]

Create runner rules

Add any runner rules for the newly created runner.

serverless-ecs

Example Configuration #

Configuration options #

Setting sensitive environment variables #

Required permissions #

auth.role_arn #

job.execution_role_arn #

job.task_role_arn #

Setup guide #

Before you begin #

Prepare the cloud environment #

Configure a state storage #

Create a runner #

Create runner rules #