<!--
.. title: Build container images in Kubernetes with kaniko
.. slug: build-container-images-in-kubernetes-with-kaniko
.. date: 2019-02-08 07:22:14 UTC
.. updated: 2019-02-08 07:22:14 UTC
.. tags: aws, docker, kubernetes, terraform
.. category:
.. link:
.. description:
.. type: text
-->

Building container images on Kubernetes is a desirable capability. It opens new
avenues of continuous integration (CI). We are no longer tied to using
something like Jenkins to schedule our CI build jobs. We can use the scheduling
proficiency of Kubernetes instead.

<!-- TEASER_END: Read more -->

There are multiple ways of building [OCI](https://www.opencontainers.org/)
container images today. The "old fashioned" method of using Docker is still
popular and improving every day. It brings the exciting concept of
[Docker-in-Docker](https://hub.docker.com/_/docker) (or DinD) to the table as
well. There's [img](https://github.com/genuinetools/img) from Genuine Tools or
[Buildah](https://buildah.io/) from Red Hat. They are all great options to get
the job done. Unfortunately, they're all very limited when it comes to building
images in Kubernetes. Enters
[kaniko](https://github.com/GoogleContainerTools/kaniko) from Google. As the
tag line for it says, "Build Container Images In Kubernetes".

The kaniko
[README](https://github.com/GoogleContainerTools/kaniko/blob/master/README.md)
on GitHub is a pretty good introduction. But as is always the case, the devil
is in the detail. That is the reason I'm writing this post.

kaniko can build a container image without needing a Docker daemon. It can run
in a Docker container (say on Docker for Mac) as well as on Kubernetes. The
aforementioned README does warn us,

> kaniko is meant to be run as an image, gcr.io/kaniko-project/executor. We do
> not recommend running the kaniko executor binary in another image, as it
> might not work.

kaniko, as of this writing, supports pushing the newly built image to
[Google GCR](https://cloud.google.com/container-registry/) and
[Amazon ECR](https://aws.amazon.com/ecr/). It supports sourcing the build
context -- Dockerfile and other files needed to build the image -- from a local
directory in the container running kaniko in addition to from either
[Google GCS](https://cloud.google.com/storage/) or
[Amazon S3](https://aws.amazon.com/s3/) bucket.

With these parameters in view, I set about running kaniko in Docker on my
localhost and in Kubernetes in [Amazon EKS](https://aws.amazon.com/eks/). My
goal was to build a container image to run
[kubectl](https://kubernetes.io/docs/reference/kubectl/overview/).

# ECR Repository and IAM

First thing was to create an ECR repository (to push the image to) and
associated IAM configuration to allow kaniko to work with it. I used Terraform
for this task.

    # ecr.tf
    provider "aws" {
    profile                 = "${var.profile}"
    shared_credentials_file = "${var.creds_file}"
    region                  = "${var.region}"
    }

    resource "aws_ecr_repository" "kubectl" {
    name = "kubectl"
    }

    resource "aws_iam_group" "ecr-power-user" {
    name = "ecr-power-user"
    path = "/"
    }

    resource "aws_iam_user" "kaniko" {
    name          = "kaniko"
    path          = "/"
    force_destroy = false
    }

    resource "aws_iam_group_membership" "ecr-power-user" {
    name = "ecr-power-user"

    users = [
        "${aws_iam_user.kaniko.name}",
    ]

    group = "${aws_iam_group.ecr-power-user.name}"
    }

    resource "aws_iam_group_policy_attachment" "kaniko_AmazonEC2ContainerRegistryPowerUser" {
    group      = "${aws_iam_group.ecr-power-user.name}"
    policy_arn = "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser"
    }

    resource "aws_iam_access_key" "kaniko" {
    user    = "${aws_iam_user.kaniko.name}"
    pgp_key = "${var.gpg_public_key}"
    }

    output "key-id" {
    value = "${aws_iam_access_key.kaniko.id}"
    }

    output "secret-key" {
    value = "${aws_iam_access_key.kaniko.encrypted_secret}"
    }

# AWS Credentials

The two output values in the above Terraform config allowed me to construct an
[AWS credentials file](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)
to be used by kaniko. For example,

    # credentials
    [default]
    aws_access_key_id=RandomIdString
    aws_secret_access_key=ChangeMeOrGoHome
    region=us-east-1

Replace the values in this file according to your requirements.

# Dockerfile

The third thing needed was a Dockerfile to build the image with kubectl
installed,

    # Dockerfile
    FROM alpine

    RUN apk add --update bash curl && \
        adduser -D -s /bin/bash me && \
        curl -sL -o /usr/local/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && \
        chmod +x /usr/local/bin/kubectl && \
        curl -sL -o /usr/local/bin/heptio-authenticator-aws https://github.com/kubernetes-sigs/aws-iam-authenticator/releases/download/v0.3.0/heptio-authenticator-aws_0.3.0_linux_amd64 && \
        chmod +x /usr/local/bin/heptio-authenticator-aws && \
        mkdir -p /home/me/.kube

    USER me

These resources were immensely helpful,

- [install-kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
- [Manage Kubernetes authentication and authorization using Heptio Authenticator and Helm](https://medium.com/nuvo-group-tech/manage-kubernetes-authentication-and-authorization-using-heptio-authenticator-and-helm-222df63d805c)

# Docker config.json

kaniko's container image comes with a default config.json, mounted in
/kaniko/.docker/, which is geared towards GCR. Since in my case I was using
ECR, I had to create a different config, like do,

    # config.json
    {
        "credHelpers": {
            "123123123.dkr.ecr.us-east-1.amazonaws.com": "ecr-login"
        }
    }

In the file above, replace _123123123_ with the account ID of your own AWS
account and replace _us-east-1_ with the region that suits you.

# Run kaniko in Docker

I first ran kaniko in Docker to learn more about how it should be used. I could
have easily run it in Kubernetes first as well.

Pull the kaniko executor image,

    $ docker pull gcr.io/kaniko-project/executor:latest

Run kaniko in Docker,

    $ docker run \
        --rm \
        -v $(PWD)/Dockerfile:/workspace/Dockerfile \
        -v $(PWD)/credentials:/root/.aws/credentials:ro \
        -v $(PWD)/config.json:/kaniko/.docker/config.json:ro \
        gcr.io/kaniko-project/executor:latest \
            --context dir:///workspace/ \
            --dockerfile Dockerfile \
            --destination 123123123.dkr.ecr.us-east-1.amazonaws.com/kubectl:latest

In the command above, replace _123123123_ with the account ID of your own AWS
account and replace _us-east-1_ with the region that suits you.

If the ~/.aws/credentials file you created above does not have a _default_
profile, you will need to provide environment variable *AWS_PROFILE* with its
value being the profile to use. I got an
error when I didn't have a default profile and I didn't set the environment
variable:
"error pushing image: failed to push to destination 123123123.dkr.ecr.us-east-1.amazonaws.com/kubectl:latest: unsupported status code 401; body;".
I believe it's related to ``/kaniko/docker-credential-ecr-login``.

I also made the mistake of mounting config.json to /root/.docker/config.json
instead of /kaniko/.docker/config.json. That caused the same error as above:
"error pushing image: failed to push to destination 123123123.dkr.ecr.us-east-1.amazonaws.com/kubectl:latest: unsupported status code 401; body;".
It wasn't easy to spot the difference between expected and actual paths because
they so closely resembled each other.


The kaniko project also provides a debug image in case you need to figure out
what's going wrong. This image was immensely helpful to me. Use the debug image
like so,

    $ docker run \
        -it \
        --entrypoint /busybox/sh \
        --rm \
        -v $(PWD)/Dockerfile:/workspace/Dockerfile \
        -v $(PWD)/credentials:/root/.aws/credentials:ro \
        -v $(PWD)/config.json:/kaniko/.docker/config.json:ro \
        gcr.io/kaniko-project/executor:debug

Notice how in the debug image I start a Bourne shell session in interative
mode instead of running the kaniko executor binary with all the other
configuration intact.

# Run kaniko in Kubernetes

Once I got a container image built with kaniko in Docker and pushed it to ECR,
I was ready to run kaniko in Kubernetes. These were the main steps,

- Create a separate namespace to isolate things
- Create a secret to store AWS credentials
- Create a configmap to store config.json
- Create a configmap to store Dockerfile
- Create a pod manifest to run kaniko

Create Kubernetes namespace manifest,

    # namespace.yml
    "kind": "Namespace"
    "apiVersion": "v1"
    "metadata":
        "name": "ns-kaniko"
        "labels":
            "name": "ns-kaniko"

Create Kubernetes namespace,

    $ kubectl apply -f namespace.yml

Create secret,

    $ kubectl --namespace ns-kaniko create secret generic aws-creds --from-file=credentials

Keep in mind that the names of the file _must_ match the name as it should
appear in the pod when the secret is mounted as a volume. I ran into issues
when I tried to have a different name for the file from which the secret is
created than the expected name in the pod. For example, if
``--from-file=credentials`` was instead ``--from-file=aws-credentials``, the
file mounted in the pod would be /root/.aws/aws-credentials instead of
/root/.aws/credentials as expected.

Create configmaps,

    $ kubectl --namespace ns-kaniko create configmap config-json --from-file=config.json
    $ kubectl --namespace ns-kaniko create configmap build-context --from-file=Dockerfile

Similar to the name of the file in secret above, the names of the file _must_
match the name as it should appear in the pod when the configmap is mounted as
a volume.

Create pod manifest, similar to the ``docker run`` command above. We want to
keep things as similar as possible so we get the same results.

    # kaniko.yml
    kind: Pod
    apiVersion: v1
    metadata:
        name: kaniko
    spec:
        containers:
            - name: kaniko
            image: gcr.io/kaniko-project/executor:latest
            args:
                - "--context=dir:///workspace"
                - "--dockerfile=Dockerfile"
                - "--destination=123123123.dkr.ecr.us-east-1.amazonaws.com/kubectl:latest"
            volumeMounts:
                - name: aws-creds
                  mountPath: /root/.aws/
                  readOnly: true
                - name: config-json
                  mountPath: /kaniko/.docker/
                  readOnly: true
                - name: build-context
                  mountPath: /workspace/
                  readOnly: true
        restartPolicy: Never
        volumes:
            - name: aws-creds
              secret:
                  secretName: aws-creds
            - name: config-json
              configMap:
                  name: config-json
            - name: build-context
              configMap:
                  name: build-context

Create kaniko pod,

    $ kubectl --namespace ns-kaniko apply -f kaniko.yml

Track the logs of this pod,

    $ kubectl --namespace ns-kaniko logs kaniko

Similar to running kaniko debug container in Docker above, we can run a debug
container in Kubernetes. Create a separate pod manifest,

    # kaniko-debug.yml
    kind: Pod
    apiVersion: v1
    metadata:
        name: kaniko-debug
    spec:
        containers:
            - name: kaniko-debug
            image: gcr.io/kaniko-project/executor:debug
            command:
                - "/busybox/cat"
            args:
                - "/workspace/Dockerfile"
                - "/root/.aws/credentials"
                - "/kaniko/.docker/config.json"
            volumeMounts:
                - name: aws-creds
                  mountPath: /root/.aws/
                  readOnly: true
                - name: config-json
                  mountPath: /kaniko/.docker/
                  readOnly: true
                - name: build-context
                  mountPath: /workspace/
                  readOnly: true
        restartPolicy: Never
        volumes:
            - name: aws-creds
              secret:
                  secretName: aws-creds
            - name: config-json
              configMap:
                  name: config-json
            - name: build-context
              configMap:
                  name: build-context

Unlike debug in Docker, here we don't start a Bourne shell to troubleshoot.
Instead, we just run the commands we need.

    $ kubectl --namespace ns-kaniko apply -f kaniko-debug.yml

Later, we can look at the pod logs,

    $ kubectl --namespace ns-kaniko logs kaniko-debug

# Conclusion

Building container images with kaniko involves a few more steps than I had
imagined before I began. I stumbled and struggled with some things, especially
when trying to debug when things went wrong. But when I got it all working it
was pretty straightforward to continue to make it work. I recommend trying this
out to see if kaniko can bring advantages to your work. It certainly has to
mine.