Docker BuildKit Deep Dive: Optimize Your Build Performance

In this article, we will explore advanced cache management in Docker builds by fully leveraging BuildKit’s capabilities. We’ll analyze its internal mechanisms and optimizations to improve build performance.

Objectives:

• Understand how BuildKit, Docker’s default build system, works.
• Analyze BuildKit’s main components and their roles.
• Deep dive into cache usage to improve build performance.

BuildKit

BuildKit is a new project under the Moby umbrella for building and packaging software using containers. It’s a new codebase meant to replace the internals of the current build features in the Moby Engine. Proposal Issue

BuildKit has become the default build method starting from Docker Engine 23.0. This means the docker build command is equivalent to docker buildx build. For earlier versions, you need to enable BuildKit by setting the DOCKER_BUILDKIT=1 environment variable.

Key points:

• docker build is equivalent to docker buildx build.
• Buildx is a CLI plugin that extends BuildKit.
• BuildKit is an independent project from Docker Engine, designed to support build methods beyond Dockerfiles.
• docker buildx build (and consequently docker build) are wrappers over BuildKit’s buildctl CLI. This wrapper is not 1:1; for example, --import-cache in buildctl becomes --cache-from in docker build.

Let’s take a look at the basic components implemented in BuildKit and how they interact.

⚠️ From now on, for convenience, we will always refer to buildctl

Client and Server

Two components are necessary for operation:

• Client: The CLI used to interact with the server, commonly via docker build, docker buildx build, or buildctl.
• Server: A daemon running as buildkitd that exposes APIs over the gRPC protocol. The daemon is already present and active by default in Docker Engine, but it can be started separately, either on your machine or in a container (we’ll see how later).

Frontend

A frontend is a component that takes a human-readable build format and converts it to LLB so BuildKit can execute it. Frontends can be distributed as images, and the user can target a specific version of a frontend that is guaranteed to work for the features used by their definition.

The frontend is responsible for retrieving the build file (e.g., Dockerfile) and converting it into a Low-Level Block (LLB) that will then be processed by the Solver.

There are currently two types of frontend: dockerfile.v0 and gateway.v0. Both interpret build definitions and convert them into LLB (Low-Level Build), but they differ significantly in features and use cases.

There’s an online tool to see how your Dockerfile is broken down into LLB: dockerfile-explorer.

dockerfile.v0

This is dedicated to interpreting Dockerfiles and is natively integrated into BuildKit as Dockerfile2LLB.

By using the # syntax= directive in your Dockerfile, you can choose the interpreter (frontend) that will parse the file. Just add the following as the first line of your Dockerfile: # syntax=docker/dockerfile:1 and run a build. In the output, you’ll see => resolve image config for docker-image://docker.io/docker/dockerfile:1. If you enter an incorrect image or tag, the build will fail. You can also specify it as an option: buildctl build --frontend gateway.v0 --opt source=docker/dockerfile ...

The docker/dockerfile image is the official BuildKit frontend. The advantage of defining the syntax directive is ensuring that the build always uses the desired interpreter version.

gateway.v0

This abstracts the concept of the interpreter, allowing any image to be used to interpret build files. It provides an abstract interface for integrating external images to generate LLB. This enables the creation of custom interpreters. The current list can be found in the project’s README.md.

Workflow

To recap, the flow executed by buildctl is as follows:

1. The buildctl client makes a gRPC call to the buildctld daemon.

2. The daemon receives the API call and forwards it to the LLBSolver (Solve Handler).

3. The LLBSolver:

   1. Creates a Job that manages the lifecycle and isolation of a single build session, acting as a container for all operations and associated resources.
   2. Creates a FrontendLLBBridge to translate the Dockerfile (or the chosen frontend) into an LLB (Low-Level Build) for each stage. The FrontendLLBBridge in turn creates the Frontend instance that performs the translation.
   3. The LLB graph produced by the Frontend is loaded into the Job. In short, an LLB is a DAG (Directed Acyclic Graph) where each node is called a Vertex. A vertex represents a single build step, such as FROM or RUN, and its dependencies on other vertices. Each vertex has a content-addressable digest representing a checksum of all graph definitions up to that vertex, including its inputs. This means that if two vertices have the same digest, they are considered identical. Vertex interface:

   type Vertex interface {
       Digest() digest.Digest
       Options() VertexOptions
       Sys() interface{}
       Inputs() []Edge
       Name() string
   }
   

   4. Executes the Solver

4. The Solver retrieves the loaded LLB from the Job and optimizes execution through vertex merging, caching, and parallelism.

   1. Each vertex is executed by a worker that retrieves operations from the Sys() method. The Solver does not know or care what operations are performed, but it records the output of the execution in the Job session.
   2. Workers run in parallel; vertices with Inputs() must wait for their dependencies to complete.
   3. Checks if the layers are present in the local cache or, if specified, in the remote cache.
   4. Communicates with the Job for progress, and errors, and build session metrics.

  sequenceDiagram
 autonumber
 participant Client as buildctl
 participant Controller as ControllerServer (buildkitd)
 participant Solver as LLBSolver
 participant Job as Job
 participant Bridge as FrontendLLBBridge
 participant Frontend as Frontend (dockerfile.v0 or gateway.v0)

 Note over Client: User exec `buildctl build`
 Client->>Controller: Solve Request (LLB Definition or Frontend)
 Controller->>Solver: Request build solving
 Solver->>Job: Create a new job
 activate Job
 Solver->>Bridge: Create FrontendLLBBridge linked to the job
 activate Bridge

 alt If request include LLB
 Bridge->>Job: Execute LLB definition
 Job-->>Bridge: Return build result
 else If a request includes Frontend definition
 Bridge->>Frontend: Exec frontend
 activate Frontend
 Frontend->>Bridge: Emit build request
 Bridge->>Job: Execute frontend build request
 Job-->>Bridge: Return results to frontend
 Frontend-->>Bridge: Return final result
 deactivate Frontend
 end

 Bridge-->>Solver: Return build result
 deactivate Bridge
 Solver->>Job: Close and cleanup job
 deactivate Job
 Solver-->>Controller: Return results to the controller
 Controller-->>Client: Return results to the client

Cache deep-dive

Now that we know the build process, let’s analyze in detail what happens when we use the cache and how we can improve our build performance. BuildKit has several types of cache:

Default cache

Layers are saved locally for the following operations:

1. source.local: Files loaded with ADD or COPY from the build context
2. exec.cachemount: Directories mounted with RUN --mount=type=cache ...
3. source.git.checkout: Source code cloned from git repositories

This is always active but can be disabled with --no-cache or by changing the buildkit configuration.

Regular

BuildKit allows loading cache layers from various sources, currently supporting:

Inline cache

Description:

The cache is embedded directly in the exported image. This allows the cache to travel with the image when pushed to a registry. Activate it with the build-arg BUILDKIT_INLINE_CACHE=1

Pros:

• Simple, no extra configuration needed.
• Integrated, cache travels with the image in the registry.

Cons:

• Limited scalability, ineffective for complex multi-stage builds.
• Increased image size, additional metadata in the image.
• Final layers only, does not include cache for intermediate stages.

Registry cache

Description:

The cache is exported as a separate image to an OCI registry (Docker Hub, ECR, GCR, etc.). Specify with --cache-to type=registry,ref=<registry>/<cache-image> and import with --cache-from type=registry,ref=<registry>/<cache-image>. Configurable for performance as needed.

Pros:

• Team sharing, ideal for CI/CD and distributed environments.
• Full cache, supports mode=max for all stages.
• Integration with external tools, compatible with any OCI registry.

Cons:

• Network latency: download/upload from the remote registry.
• Storage costs: potential costs for private registries.

Filesystem cache

Description:

The cache is exported/imported from a local directory in OCI image layout format. Use with --cache-to type=local,dest=./cache and --cache-from type=local,src=./cache. Useful for builds on shared filesystems or local testing.

Pros:

• Full control: manual data management.
• Air-gapped friendly: useful in isolated environments.
• Simple backup: copy directory for archiving.

Cons:

• Manual management: requires scripts for cleanup/backup.
• Not scalable: inconvenient for large teams.
• Non-optimized format: takes up more space than registry format.

Cache GHA, S3, Azure Blob

This is an experimental feature. I have yet to explore it and have no particular suggestions, but it is an option nevertheless.

Advantages

Speed and repeatability of builds across different environments

An additional cache (e.g., remote on registry, directory, or GitHub Actions) allows reusing already built layers even on different machines or CI/CD pipelines, drastically reducing build times when the local cache is unavailable. Layers are only downloaded when needed.

Greater cache resilience and persistence

In ephemeral environments (like CI/CD), the local cache is deleted between jobs. An external cache ensures layers are not lost and are always available for future builds, even after cleanups or host changes.

Flexibility and advanced optimization

You can combine multiple caches (e.g., caches from different branches or main and feature branches), choosing the best strategy for your workflow. BuildKit can consult both local and one or more remote caches, retrieving layers wherever available and optimizing both time and resources.

Min vs Max

• mode=min (default):
  • Saves only the final layers that make up the output image (those exported in the final image).
  • Cache export time is very low.
  • Re-build time varies depending on cache hit ratio, but is generally slower.
  • Useful for builds that don’t often modify intermediate stages, when registry/local space is critical, and to reduce upload times to remote registries.
• mode=max:
  • Saves all layers, including those from intermediate stages not included in the final result (e.g., test or intermediate build stages in a multi-stage Dockerfile).
  • Cache export time is much higher depending on the number and size of intermediate layers.
  • Re-build time varies depending on the cache hit ratio, but generally faster.
  • Useful when there are many intermediate stages (e.g., build, test, lint) that rarely change, and for sharing cache between branches.

Configurations

Some configurations should be considered regarding the cache.

Metadata

Some registries, such as AWS ECR, do not support certain OCI metadata. To export, you must include the options image-manifest=true and oci-mediatypes=true. Full command:

--cache-to type=registry,ref=<ecr-registry>:<tag>,image-manifest=true,oci-mediatypes=true

Compression

Cache export allows you to choose the compression method for layer contents. The default is gzip, but zstd, estargz, and uncompressed are also supported. You can also choose the compression level: the higher it is, the more compressed, but execution time may increase (0-9 for gzip and estargz, 0-22 for zstd).

Full command:

--cache-to type=registry,ref=<ecr-registry>:<tag>,compression=zstd,compression-level=3

Hands On

GitHub Repository: Docker BuildKit Example

It’s time to get hands-on and try buildctl directly. I’ve created a GitHub repository with all the code presented below, easily clonable and usable to follow this exercise step-by-step.

Goals

• Run the buildkitd daemon locally
• Use the buildctl CLI to create Docker images
• Understand layer logic
• Make the best use of the cache

Setup

Requirements

You need to have the Docker engine installed on your machine (I recommend version ≥ 23.0).   I’m currently using Docker version 28.0.1, build 068a01e. To check your version, simply run docker --version.

For installing buildctl, refer to the complete official guide available at github.com/moby/buildkit.

Repository structure

The repository is organized as follows:

• Makefile: contains the buildctl commands mapped to their respective targets. We’ll use these for our tests and to facilitate the exercise.
• app/: contains our application, i.e., a Dockerfile and two test files. These will help us understand layer logic.
• .cache/: will contain the cache layers. It’s not versioned and will only be visible after running the commands on your machine.
• .images/: contains the built images in .tar format.

Once all requirements are installed and the repository is cloned, we’re ready to start our buildkitd daemon. Remember, Docker has it active by default, but the goal is to create and manage our daemon independently.

Let’s Go

1. Run Buildkit daemon

The Makefile automatically creates the BuildKit daemon as a Docker container. This is just to make local installation easier.

Let’s get to know the CLI by running these commands:

• make init creates the BuildKit daemon container and sets up the project files/folders.
• buildctl debug info to get info about the activated daemon, but at this stage you’ll get the error

error: failed to call info: Unavailable: connection error: desc = "transport: Error while dialing: dial unix /var/run/buildkit/buildkitd.sock: connect: no such file or directory"

• export BUILDKIT_HOST=docker-container://buildkitd and then run buildctl debug info again, this time successfully.
• Make sure there are no images present in BuildKit with buildctl du -v --format json. The response should be null.

⚠️ As an alternative to the Docker container, you can use lima and create a virtual machine from the official BuildKit template with limactl start template://buildkit. To ensure the Makefile targets use the correct socket, run make BUILDKIT_HOST="unix://$HOME/.lima/buildkit/sock/buildkitd.sock" or permanently modify the Makefile. For the commands above, the console export should have the lima socket path.

2. Dockerfile inspect

The Dockerfile is written specifically to demonstrate how cache layers work and how to optimize them. In detail:

• Two stages, a build stage and a final stage
• Loads the test-file, a 1 GB file created during init
• Loads requirements files containing dependencies

3. Build app without cache

Our goal is to understand how the build works and how the cache affects performance. We’ll test the difference between the min and max types to see when to use each.

Note: The cache will be written locally and then extracted so that we can analyze the differences in detail.

Proceed with:

1. make build-min builds the Dockerfile with min cache type. In the output, you’ll notice that preparing build cache for export takes between 0.5 and 2 seconds.
2. make build-max builds the same Dockerfile, but with max cache type. The first thing you’ll notice is that preparing build cache for export takes between 20 and 30 seconds.
3. make cache-compare compares the two caches.

The first thing that stands out is the difference in export time. But, by comparing, you’ll also see that:

• The .cache/app/max-extracted directory is 1GB larger than the min type.
• The max cache has 16 layers, while min has only 11.
• The test-file is only present in the max cache at .cache/app/max-extracted/<layer-digest>/app/test-file
• Total build time: max 50 seconds vs 25 seconds for min

These data show how min and max have different logics and how they impact performance, not only in terms of time but also storage and computation.

As seen earlier, the max mode cache saves all layers from all Dockerfile stages, so subsequent builds have all layers ready to use unless invalidated by changes.

However, this advantage can become a problem with increased build times when the cache is invalid or missing.

In our case, we have a 1GB test-file (simulating a realistic build stage) in the builder stage, which is saved in the max cache, causing a 25-second increase in cache export time.

Now imagine a real case, such as a Node or AI build with many packages and libraries occupying several GBs, and possibly multiple stages. Here, max mode may be the wrong or at least less efficient choice.

Performance

Dockerfile refactoring

To improve performance, you need to analyze the Dockerfile and ensure that the layers you cache are arranged so that a previous layer doesn’t invalidate the next. In our case, at line 12 we copy the first requirements.txt and at line 16 the requirements2.txt. This means:

In max mode:

• If requirements.txt changes, the builder stage cache layer is invalidated, so a new build will rerun the builder stage.
• If requirements.txt doesn’t change, all cache layers are valid.

In min mode:

• If requirements.txt changes, the builder stage cache layer is invalidated, so a new build will rerun the builder stage.
• If requirements.txt doesn’t change, all builder cache layers are valid.

This simple example shows that in this case, using max mode doesn’t make sense because any change will invalidate the cache layers in the same way, forcing a new build, but with the added cost of longer export times and more storage.

Of course, this is a simple case, but it demonstrates that the choice of cache mode depends strictly on how the Dockerfile is written. To increase cache performance, it’s important to define layers correctly, grouping and ordering them so that frequently changing ones are at the end.

Compression type

Compression plays a key role in cache export. In a real case, we had a pipeline build for an AI image, which with an invalid cache took 13-14 minutes.

Analyzing why led to this article. In short, by refactoring the Dockerfile and switching from gzip to zstd compression, cold cache build times dropped to 5 minutes.

This is because, according to the OCI/Docker spec, each image layer is a tar archive containing the files for that layer. By default, when exporting an image (to a tar file or registry), each layer is compressed with gzip.

BuildKit currently supports gzip, zstd, estargz, and uncompressed.

Comparing compression types, zstd is significantly faster than others, both for compression and decompression and has a better compression ratio.

This leads to reduced export times and smaller cache layers. Here is a great article with benchmarks on the topic.

Cache from

Another important aspect is the cache layer selection strategy. An effective technique is to use multiple --cache-from sources for each build:

• From the latest or main image to ensure unchanged layers are present.
• From the commit sha or branch name when working on feature branches to ensure cache layers are always up to date.

As mentioned earlier, layers are only downloaded when needed, so if a layer is cached in multiple images, it won’t be downloaded more than once.

Cache to

The --cache-to option must be used carefully, as each definition corresponds to an upload to the registry, meaning longer execution times, and higher storage and networking costs.

Extras

When you build with buildctl, images are not automatically stored in the Docker daemon (unlike docker build). This is because BuildKit is designed to be independent of Docker’s runtime and uses its own cache and storage system.

Where are images saved?

Images resulting from a BuildKit build are saved in an internal content store, accessible only by BuildKit unless explicitly exported. This store is located at:

• Linux: BuildKit data, including snapshots, cache, and metadata, are saved in /var/lib/buildkit
• MacOS Docker Desktop: On MacOS, Docker (and thus BuildKit) data are saved on the VM’s “disk image” and are not directly accessible.

If you look for the image with docker images, you won’t find it because it hasn’t been loaded into the Docker daemon.

How to export the image to Docker?

To make the image visible to Docker, you must explicitly export it with --output type=docker or --output type=registry:

1. Export as Tar and then load (method used in the example code)

To save the image as a .tar file and then load it into Docker:

buildctl build \
 --frontend=dockerfile.v0 \
 --local context=. \
 --local dockerfile=. \
 --output type=oci,dest=image.tar

Then load it into Docker with: docker load -i image.tar

2. Export directly to Docker

To load the image directly into the Docker daemon:

buildctl build \
 --frontend=dockerfile.v0 \
 --local context=. \
 --local dockerfile=. \
 --output type=docker | docker load

This command creates the image and pipes it directly to docker load, making it visible in docker images.

3. Push the image to a registry

To upload the image directly to a registry (e.g., Docker Hub or a private registry):

buildctl build \
 --frontend=dockerfile.v0 \
 --local context=. \
 --local dockerfile=. \
 --output type=registry,name=myrepo/myimage:latest

Now the image will be available via docker pull myrepo/myimage:latest.

Conclusions

To recap this deep dive, to improve build performance you need to consider several factors: write an efficient Dockerfile, have a cache strategy, choose the cache mode appropriate for your Dockerfile and context, and select zstd for compression.

1. BuildKit: We explored BuildKit’s processes in detail, looking closely at its components and the build process flow.
2. Cache understanding: We saw how the cache works and the differences between min and max. A correct configuration can improve build and storage performance.
3. Dockerfile optimization: Strategically arranging layers and stages is key to maximizing build and cache efficiency.
4. Compression: Choosing the right compression method is crucial, and with zstd you can significantly reduce export times compared to gzip.
5. Distributed cache: Using cache (registry or filesystem) improves resilience and speed of builds in CI/CD environments.

Resources

• https://depot.dev/blog/buildkit-in-depth

• https://drive.google.com/file/d/1zGMQipL5WJ3sLySu7gHZ_o6tFpxRXRHs/view

• https://docs.google.com/presentation/d/1tEnuMOENuoVQ3l6viBmguYUn7XpjIHIC-3RHzfyIgjc/edit?slide=id.g11ca940f0ee_0_15#slide=id.g11ca940f0ee_0_15

• https://crazymax.dev/buildkit/

• https://pkg.go.dev/github.com/moby/buildkit/client/llb#WriteTo

• https://www.augmentedmind.de/2023/10/29/docker-and-buildkit-3-approaches/

• https://docs.docker.com/build/buildkit

• https://github.com/moby/buildkit

• https://github.com/moby/buildkit/tree/master/docs

• https://github.com/moby/buildkit/blob/648ec56d9b65b74fb8bfca6a8d966238f5d39bcf/frontend/dockerfile/docs/reference.md