Scanning and remediating vulnerabilities with Grype

Date: 2023-08-19

The source code for this lab exercise is available on GitHub.

Consider our typical DevSecOps CI/CD pipeline that triggers automated unit and integration testing, container image building, vulnerability scanning, image pushing and signing, all the way up to deploying to a properly secured production environment on every developer commit to a Git repository.

DevSecOps CI/CD pipeline

We’ve seen how to construct a complete DevOps CI/CD pipeline with GitHub Actions, how container image signing and verification can be achieved with Sigstore Cosign and policy-controller, but we’ve yet to explore some of the available tools to perform vulnerability scanning on container images and how to make use of the feedback provided by these tools to start securing our applications and microservices.

In the lab to follow, we’ll see how vulnerability scanning can be conveniently achieved with Grype and how various systematic techniques can be applied to start securing our microservices at the container image level.

Lab: Minimizing container image vulnerabilities for a Rocket microservice

Prerequisites

Familiarity with Linux, Docker and containers is assumed. If not, consider following through the official Docker guide. You may also wish to enroll in LFS101x: Introduction to Linux, a self-paced online course offered by The Linux Foundation on edX at no cost.

Setting up your environment

You’ll need a Linux environment with at least 2 vCPUs and 4G of RAM. The reference distribution is Ubuntu 22.04 LTS, though the lab should work on most other Linux distributions as well with little to no modification.

We’ll set up the following tools:

Installing Docker

Docker (hopefully) needs no introduction - simply install it from the system repositories and add yourself to the docker group:

sudo apt update && sudo apt install -y docker.io
sudo usermod -aG docker "${USER}"

Log out and back in for group membership to take effect.

Installing Grype

Grype is an open source vulnerability scanning tool for container images developed and maintained by Anchore, dedicated to improving software supply chain security.

Install it using the official instructions but we’ll install it under $HOME/.local/bin/ so no sudo is required:

curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b "$HOME/.local/bin"

Log out and back in for Grype to appear in your PATH.

Verifying everything is installed correctly

Run the following commands to check the version of each tool we just installed:

docker --version
grype version

Sample output:

Docker version 20.10.25, build 20.10.25-0ubuntu1~22.04.1
Application:          grype
Version:              0.65.2
Syft Version:         v0.87.1
BuildDate:            2023-08-17T20:03:30Z
GitCommit:            51223cd0b1069c7c7bbc27af1deec3e96ad3e07d
GitDescription:       v0.65.2
Platform:             linux/amd64
GoVersion:            go1.19.12
Compiler:             gc
Supported DB Schema:  5

Evaluating base image selection and image build strategy from a security perspective using Grype

Let’s consider a Rocket microservice exposing the following API:

The microservice is written in Rust using the Rocket framework, though developing a RESTful microservice with Rocket isn’t the main focus of this lab so we’ll skip over the implementation details - interested Rustaceans may inspect the source code at their own leisure.

Clone the project locally with Git and navigate to the project root:

git clone https://github.com/DonaldKellett/rocket-date-server.git
pushd rocket-date-server/

Now inspect the contents of the default Dockerfile shown below:

FROM rustlang/rust:nightly-bookworm
WORKDIR /app
COPY src/ /app/src/
COPY Cargo.toml /app
COPY Cargo.lock /app
RUN cargo install --path .
CMD ["/usr/local/cargo/bin/rocket-date-server"]

As usual, we select a suitable base image, copy our application source code to the image, build the application using the right tools and specify a suitable default command to run whenever a container is created from our image.

Let’s analyze our selection of the base image rustlang/rust:nightly-bookworm:

We expect this image to be reasonably secure for most typical real-world scenarios. But anyway, let’s first build our image:

docker build -t rocket-date-server:0.1.0 .

Now spin up a container based on our newly built image:

docker run \
    --rm \
    -d \
    -p 8000:8000 \
    --name rocket-date-server \
    rocket-date-server:0.1.0

Our microservice listens on port 8000 locally - let’s give it a test drive to confirm that it is operational:

curl localhost:8000/date
curl localhost:8000/version

Sample output:

{"date":"2023-08-19 06:22:44"}

{"version":"0.1.0"}

Now stop and delete the container to conserve resources:

docker stop rocket-date-server

As mentioned above, our image is using up-to-date software so we expect it to be reasonably secure. But how secure is it really? Let’s find out by scanning our image with Grype:

grype rocket-date-server:0.1.0

Using a default image with up-to-date software

Turns out there are 729 known vulnerabilities in total, 2 of which are critical and 45 of which are high. That’s a lot of vulnerabilities!

Since the default image contains a lot of unneeded software, the attack surface is large and it is easy to discover new vulnerabilities which may not have a known immediate patch. If we really care about security, we’ll have to do better.

Now consider a modified Dockerfile Dockerfile.slim based on a slimmed down Debian image rustlang/rust:nightly-bookworm-slim with less unneeded software:

FROM rustlang/rust:nightly-bookworm-slim
WORKDIR /app
COPY src/ /app/src/
COPY Cargo.toml /app
COPY Cargo.lock /app
RUN cargo install --path .
CMD ["/usr/local/cargo/bin/rocket-date-server"]

Build a new image from our modified Dockerfile, giving the image a tag of 0.1.0-slim this time to distinguish it from our original image:

docker build -f Dockerfile.slim -t rocket-date-server:0.1.0-slim .

Now spin up a container from our new image. Other than the updated tag name 0.1.0-slim, the command used should be otherwise identical:

docker run \
    --rm \
    -d \
    -p 8000:8000 \
    --name rocket-date-server \
    rocket-date-server:0.1.0-slim

Run the same curl commands again to confirm our modified image is operational. The exact curl commands are elided and not repeated here for brevity.

Now stop our container again:

docker stop rocket-date-server

Give our slimmed-down image a scan:

grype rocket-date-server:0.1.0-slim

Using a slimmed-down image with up-to-date software

Notice how the vulnerability count for our slimmed-down image is reduced to 255 with no critical vulnerabilities and “only” 14 high vulnerabilities. While still far from ideal, this is already a great step forward from our initial condition. So remember - if you care about the security of your applications at all, start by replacing the default bloated base image with a slimmed-down image containing less unneeded software and thus a reduced attack surface.

But why stop here when we can go further? Let’s explore Google’s distroless images and see how they can help us further reduce the attack surface of our microservices.

Consider the following Dockerfile Dockerfile.distroless:

FROM rustlang/rust:nightly AS build-env
WORKDIR /app
COPY src/ /app/src/
COPY Cargo.toml /app
COPY Cargo.lock /app
RUN cargo install --path .

FROM gcr.io/distroless/cc:nonroot
WORKDIR /app
COPY --from=build-env /usr/local/cargo/bin/rocket-date-server /app
CMD ["/app/rocket-date-server"]

Unlike our previous Dockerfiles, the base gcr.io/distroless/cc:nonroot for our application image is specified on line 8, which contains only a bare minimal Debian base system with a minimal C runtime required to run most typical simple applications. It does not even contain the APT package manager so you cannot install development dependencies and build your application directly from there.

Notice line 1 of our Dockerfile specified as follows:

FROM rustlang/rust:nightly AS build-env

This specifies a different base image rustlang/rust:nightly containing all the usual development dependencies for building our application from source code. Then, once our application is successfully built, we copy only the application binary and possibly its runtime dependencies to our distroless base image to be shipped to production, without including unnecessary artifacts such as the application source code and development dependencies - this ensures we keep the attack surface of our microservice at a minimum. This method of building and preparing our application image is known as a multi-stage build.

Now build our image from this Dockerfile and tag it 0.1.0-distroless:

docker build -f Dockerfile.distroless -t rocket-date-server:0.1.0-distroless .

You should test the image again to confirm it is functional, the exact commands of which are elided for brevity.

Now scan our distroless image for known vulnerabilities:

grype rocket-date-server:0.1.0-distroless

Basing our microservice on a distroless image

Notice how the vulnerability count is now reduced to just 15, with no critical or high vulnerabilities and just 4 medium vulnerabilities. This is a staggering improvement from our previous “slimmed-down” image! Distroless works especially well for application binaries that do not require a full-fledged language runtime such as Python or Node.js from a security perspective, so keep that in mind when deciding whether to base your application images on Distroless.

Before we conclude this lab, let’s look at one more option to base our image upon: Alpine Linux.

Alpine Linux is a security-oriented, lightweight Linux distribution based on musl libc and busybox.

So basically, it’s designed to be small, fast and secure. The drawback is that its choice of musl libc as opposed to the usual GNU C library glibc implies subtle behavioral differences from most other Linux distributions that might matter for larger, more complex applications and that applications compiled for Alpine often cannot be executed directly on most other Linux distributions and vice-versa.

Here’s our final Dockerfile Dockerfile.alpine for this lab:

FROM rustlang/rust:nightly-alpine AS build-env
RUN apk add -U musl-dev
WORKDIR /app
COPY src/ /app/src/
COPY Cargo.toml /app
COPY Cargo.lock /app
RUN cargo install --path .

FROM alpine
WORKDIR /app
COPY --from=build-env /usr/local/cargo/bin/rocket-date-server /app
CMD ["/app/rocket-date-server"]

Again, we use a multi-stage build to ensure that our final image contains exactly what is required to run our application.

Now build our image and give it a tag of 0.1.0-alpine:

docker build -f Dockerfile.alpine -t rocket-date-server:0.1.0-alpine .

Give it a test run again to ensure the application works as expected, then scan the image for vulnerabilities:

grype rocket-date-server:0.1.0-alpine

Using Alpine as base image for our microservice

This time, Grype reported no known vulnerabilities - the best we could possibly achieve and a final step up from our distroless-based application image. So remember this folks - use an updated version of Alpine with a multi-stage build for maximal security for your microservices where it really matters ;-)

Conclusion

We’ve covered how container image vulnerability scanning is an integral component of every typical DevSecOps CI/CD pipeline, how it works in practice by using ready-made open source tools such as Grype available at no cost and ways to minimize microservice vulnerabilities by selecting the correct base image for each workload as well as using a multi-stage build to ensure only the necessary artifacts end up in the final application image thus minimizing its attack surface.

I hope you enjoyed this article and stay tuned for more content ;-)

Subscribe: RSS Atom [Valid RSS] [Valid Atom 1.0]

Return to homepage