DEVOPS TRIBUNE ENGINEERING INTELLIGENCE FOR THE REAL WORLD
CONTAINERS & RUNTIMES · 23 MIN READ
Docker official logo

Docker Container Architecture: Multi-Stage Builds, Security & Networking

An in-depth platform engineering guide to Docker container architecture: Linux namespaces, cgroups v2 resource controllers, OCI runtime boundaries, multi-stage build optimization, non-root security hardening, and production networking.

By DevOps Tribune Engineering Staff · Published November 17, 2026

In modern cloud-native systems, containers represent the universal packaging and execution unit for software. However, containers are frequently misunderstood as lightweight virtual machines. In reality, a container is not a virtualization boundary: there is no hypervisor, no guest operating system kernel, and no virtual hardware abstraction. A container is simply a standard Linux process running directly on the host kernel, restricted by native kernel primitives: namespaces for isolation and control groups (cgroups) for resource metering.

When engineering teams treat containers like virtual machines—stuffing entire operating system distributions, package managers, compilers, and root privileges into production images—they introduce massive security attack surfaces, bloated image registries, and slow deployment cycles. A 1.2 GB container image containing apt-get, curl, and build compilers provides malicious actors with convenient reconnaissance tools if remote code execution occurs. Conversely, a production-hardened container built via multi-stage compilation onto a minimal distroless or scratch base image reduces the binary footprint to under 40 MB, strips all operating system utilities, drops root capabilities, and renders the filesystem immutable.

This comprehensive technical guide examines the deep systems architecture of Docker containers. We dissect the Linux kernel mechanisms underpinning container isolation, map the Open Container Initiative (OCI) runtime stack across containerd and runc, detail advanced multi-stage build caching strategies, configure rootless daemon execution, trace bridge and overlay networking packet flows, and provide complete, copy-paste-ready hardened production configurations.


1. Linux Container Mechanics: Namespaces and cgroups v2

The foundation of containerization resides entirely within the Linux kernel. Docker orchestrates two distinct kernel subsystems to construct the container boundary:

1. Linux Namespaces (What the Process Can See)

Namespaces partition kernel resources such that a group of processes sees a completely isolated view of the operating system:

  • PID Namespace (Process Isolation): Isolates the process ID space. The primary container process runs as PID 1 inside the container namespace, while on the underlying host, it operates as a standard high-numbered PID (e.g., PID 28412). When the container's PID 1 terminates, the kernel terminates all child processes within that namespace.
  • NET Namespace (Network Isolation): Provides an independent virtual network stack, complete with its own loopback interface (lo), virtual ethernet interfaces (veth), routing tables, and iptables filtering rules.
  • MNT Namespace (Mount Isolation): Isolates filesystem mount points. Combined with pivot_root, the container only sees its own root filesystem (rootfs), preventing access to the host's /etc, /var, or root storage.
  • IPC Namespace (Inter-Process Communication): Isolates System V IPC and POSIX message queues, preventing containers from snooping on shared memory segments used by host processes.
  • UTS Namespace (Host & Domain Isolation): Allows the container to declare its own hostname and domain name without altering the underlying host machine.
  • USER Namespace (Root Mapping): Maps the container's internal UID/GID space to non-privileged UIDs on the host. An application running as root (UID 0) inside the container is mapped to an unprivileged user (e.g., UID 100001) on the host, preventing host root escalation.

2. Control Groups / cgroups v2 (What the Process Can Use)

While namespaces dictate visibility, cgroups enforce strict resource consumption boundaries. Modern Linux distributions (RHEL 9, Ubuntu 22.04+, Debian 12) utilize cgroups v2, providing unified resource hierarchy controllers:

  • Memory Controller (memory.max, memory.high): Implements hard memory limits. When a container's resident set size (RSS) memory exceeds memory.max, the Linux Out-Of-Memory (OOM) killer immediately terminates the process with exit code 137.
  • CPU Controller (cpu.max): Implements bandwidth quota enforcement over a 100ms CFS quota period. If a container is assigned 200000 100000 (2 CPUs), it is allotted 200ms of CPU time across all threads every 100ms wall-clock window.
  • I/O Controller (io.max): Throttles read and write IOPS and bytes per second against block devices, preventing noisy neighbor processes from saturating NVMe or SSD disk buses.

2. The OCI Runtime Architecture: Docker Engine, containerd, and runc

In modern enterprise environments, Docker is not a monolithic binary. Following the Open Container Initiative (OCI) specifications, container execution is split into modular layers:

[ Docker CLI / API client ]
             |
             v  (gRPC / REST)
[ Docker Daemon (dockerd) ]  -- Manages networks, volumes, buildkit, image layers
             |
             v  (gRPC over /run/containerd/containerd.sock)
[ containerd (Core Runtime) ] -- Manages image distribution, snapshots, execution lifecycle
             |
             v  (fork/exec)
[ containerd-shim ]          -- Decouples running container from daemon restarts
             |
             v  (libcontainer)
[ runc (Low-Level OCI Runtime) ] -- Invokes Linux clone(), unshare(), setns() syscalls
             |
             v
[ Running Container Process ]

Why the containerd-shim Layer Matters to SREs

The containerd-shim process serves a crucial reliability function in production infrastructure: it holds open the container's standard I/O streams (stdin, stdout, stderr) and exit code status pipes. Because the shim is the direct parent of the container process rather than the Docker daemon, you can restart, upgrade, or crash the dockerd daemon without terminating or interrupting running production containers (known as Live Restore in /etc/docker/daemon.json).


3. Multi-Stage Dockerfile Engineering: Slimming Images from 1.2 GB to 35 MB

Building production container images requires separating the build-time environment (SDKs, compilers, package managers, dev headers) from the runtime environment (compiled binary and minimal dependencies).

Comparing Image Base Types

Base Image Typical Size Shell Included? C Library Production Security Posture
ubuntu:24.04 / debian 80 - 120 MB Yes (bash, sh) glibc High CVE surface; contains package managers and system tools
alpine:3.20 7 MB Yes (ash) musl libc Low CVE surface; potential musl/glibc C-extension performance quirks
gcr.io/distroless/static 2 - 20 MB No (Zero shell) None / static Optimal; zero shell, zero package manager, minimal CVE footprint
scratch 0 MB No None Absolute Zero Surface; requires statically compiled binary (Go/Rust)

Optimizing Layer Caching Mechanics

Docker executes Dockerfile instructions as ordered layers. If an instruction invalidates the cache, all subsequent layers must rebuild from scratch. The golden rule of Dockerfile optimization is: order instructions from least-frequently changed to most-frequently changed.

# ANTI-PATTERN: Copies all source code before installing dependencies
COPY . .
RUN npm install # Invalidate cache on every single line of code change!

# PRODUCTION PATTERN: Cache package manifests separately
COPY package.json package-lock.json ./
RUN npm ci --only=production # Cached permanently until dependencies change!
COPY src/ ./src/

4. Production Hardened Multi-Stage Dockerfile Template

The following production multi-stage Dockerfile demonstrates an enterprise Go/Node microservice build. It compiles in a heavy SDK container, extracts the static binary into a distroless runtime container, enforces non-root UID 10001, strips debugging symbols, and includes an explicit container health check:

# Stage 1: Build & Compilation Environment
FROM golang:1.23-alpine AS builder

# Install build dependencies and security certificates
RUN apk add --no-cache git ca-certificates tzdata

WORKDIR /build

# Copy dependency manifests first to leverage Docker layer caching
COPY go.mod go.sum ./
RUN go mod download && go mod verify

# Copy application source code
COPY . .

# Build statically linked binary:
# CGO_ENABLED=0 disables C bindings, generating a portable static binary
# -ldflags="-s -w" strips symbol tables and debug information (-30% size)
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build     -trimpath     -ldflags="-s -w -X main.version=1.4.2"     -o /build/api-server ./cmd/server

# Stage 2: Hardened Production Runtime Environment
FROM gcr.io/distroless/static-debian12:nonroot

WORKDIR /app

# Import CA certs and timezone data from builder
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /usr/share/zoneinfo /usr/share/zoneinfo
COPY --from=builder /build/api-server /app/api-server

# Expose internal HTTP service port
EXPOSE 8080

# Run strictly as unprivileged non-root user (UID 65532 nonroot)
USER nonroot:nonroot

# Execute the application binary
ENTRYPOINT ["/app/api-server"]

5. Docker Container Networking Architecture

Docker provides five distinct network drivers that govern packet routing between containers and the external network:

1. Bridge Networking (Default Single-Host Mode)

When Docker starts, it creates a Linux software bridge interface on the host named docker0 (typically assigned subnet 172.17.0.1/16). When a container launches in bridge mode:

  1. The kernel creates a virtual ethernet pair (veth). One end is attached to the docker0 bridge on the host; the other end is moved into the container's private NET namespace as eth0.
  2. Outbound traffic from the container to the public internet undergoes Source Network Address Translation (SNAT / IP Masquerade) via host iptables.
  3. Inbound traffic to published ports (-p 8080:80) undergoes Destination NAT (DNAT), forwarding packets arriving on host port 8080 to the container's private IP.

2. User-Defined Bridge Networks vs Default Bridge

In production, never use the default bridge network. User-defined bridges provide two critical architectural advantages:

  • Automatic DNS Service Discovery: Containers on a custom bridge resolve each other by container name (e.g. curl http://postgres-db:5432) via an internal DNS server embedded at 127.0.0.11. The default bridge does not support automatic DNS.
  • Network Isolation: Containers on different custom bridges cannot communicate with each other, isolating database tiers from public web tiers.
# Creating an isolated production bridge network
$ docker network create --driver bridge --subnet 172.28.0.0/16 production-net

# Running containers attached to the private network
$ docker run -d --name redis-cache --network production-net redis:7-alpine
$ docker run -d --name api-service --network production-net -p 8080:8080 my-api

6. Storage Primitives: Bind Mounts, Named Volumes & tmpfs

Because container root filesystems are ephemeral (backed by an overlay2 copy-on-write storage driver), data written inside a container is lost when the container is deleted. Docker provides three distinct storage mechanisms:

Mount Type Host Location Performance & Isolation Production Recommendation
Named Volumes Managed by Docker in /var/lib/docker/volumes/ Native filesystem speed; completely managed by Docker API; decoupled from host paths Best practice for databases (Postgres, MySQL, Redis state)
Bind Mounts Arbitrary path on host (e.g. /opt/data) Depends on host file permissions; exposes host filesystem to container Ideal for configuration files (e.g. nginx.conf) and local dev hot-reloading
tmpfs Mounts Host RAM / swap space only Ultra-fast memory speed; zero disk persistence; data deleted on container stop Ideal for sensitive in-memory scratch space, encryption keys, and session tokens
# Production Docker Compose demonstrating volume and tmpfs segregation
services:
  database:
    image: postgres:16-alpine
    volumes:
      - pgdata:/var/lib/postgresql/data # Persistent named volume
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql:ro # Read-only bind mount
    tmpfs:
      - /tmp # Memory-backed scratch space
volumes:
  pgdata:
    driver: local

7. Container Security Hardening: Rootless Execution & Capabilities

In default installations, the Docker daemon runs as root on the host system. If a container process breaks out of its namespace due to a Linux kernel zero-day vulnerability (such as Dirty COW or runc CVE-2024-21626), the attacker gains full root control over the physical host. Production systems enforce defense-in-depth security controls:

1. Dropping Unnecessary Linux Capabilities

By default, Docker grants containers 14 Linux capabilities (e.g., CAP_NET_RAW, CAP_CHOWN, CAP_MKNOD). Production workloads should drop all capabilities and selectively grant only what is required:

# Run container with all capabilities dropped except port binding
$ docker run -d     --cap-drop=ALL     --cap-add=NET_BIND_SERVICE     --read-only     --tmpfs /tmp:rw,noexec,nosuid     my-hardened-app

2. Read-Only Root Filesystems

Running a container with --read-only prevents any modifications to the root filesystem. If an attacker discovers a remote code execution vulnerability, they cannot download remote malware binaries, modify system libraries, or install backdoors. Any required scratch directories are explicitly mounted as temporary in-memory tmpfs volumes.


8. Advanced BuildKit Acceleration: Compiler Cache Mounts & Remote Caching

In high-velocity CI/CD engineering teams, waiting 8 to 15 minutes for containers to compile on clean runner instances destroys developer productivity. Even with multi-stage Dockerfiles, standard Docker builds discard compiler intermediate caches (such as Go build caches, Rust cargo targets, or Node.js npm caches) between runs because the container filesystem is ephemeral. BuildKit solves this via specialized cache mount primitives.

1. Ephemeral Compiler Cache Mounts

Using BuildKit's --mount=type=cache instruction, you can mount a persistent cache directory directly into the compilation step without baking those intermediate artifacts into the final image layer:

# syntax=docker/dockerfile:1.7
FROM golang:1.23-alpine AS builder
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download

COPY . .

# Cache Go module downloads and compiler build cache across independent builds
RUN --mount=type=cache,target=/go/pkg/mod     --mount=type=cache,target=/root/.cache/go-build     CGO_ENABLED=0 go build -trimpath -o /out/server ./cmd/server

When running in local development or on dedicated CI nodes, BuildKit preserves the contents of /root/.cache/go-build across builds. Subsequent image builds complete in 2 to 5 seconds instead of minutes.

2. Distributed Remote Caching in GitHub Actions and GitLab CI

On ephemeral cloud runners where disk state is wiped after every workflow, BuildKit supports exporting image layer caches directly to container registries (such as AWS ECR, GitHub Container Registry, or Docker Hub) using inline or dedicated cache manifests:

# Building with distributed remote registry caching
$ docker buildx build     --platform linux/amd64,linux/arm64     --cache-from type=registry,ref=ghcr.io/devopstribune/core-api:buildcache     --cache-to type=registry,ref=ghcr.io/devopstribune/core-api:buildcache,mode=max     --push     --tag ghcr.io/devopstribune/core-api:1.4.2 .

9. Storage Internals: The Overlay2 Filesystem Mechanics

Understanding how Docker writes files requires inspecting the Linux OverlayFS (overlay2) union mount driver. An overlay filesystem merges multiple discrete directories into a unified view:

  • lowerdir (Read-Only Image Layers): Every line in a Dockerfile that modifies files creates an immutable, read-only layer stacked on top of previous layers. Multiple running containers share the exact same lowerdir storage on disk, consuming zero redundant storage.
  • upperdir (Container Read-Write Layer): When a container launches, Docker allocates a thin, dedicated read-write directory on the host host. Any newly created files reside here.
  • merged (Unified Container View): The kernel presents the union of lowerdir and upperdir as the container's root directory.
  • Copy-on-Write (CoW) Performance Penalty: When an application modifies an existing file that originates from a lower image layer, the Linux kernel must first duplicate the entire file from lowerdir up into upperdir before applying the write. For heavy random-write workloads (such as relational databases or Elasticsearch indexes), this introduces severe I/O latency penalties, which is why stateful databases must always store data on dedicated Docker Volumes that bypass the overlay2 driver.

10. Automated Container Vulnerability Scanning in CI/CD

Embedding vulnerability scanners into CI/CD pipelines ensures that vulnerable base images or outdated dependencies are rejected before images push to production registries. Standard automated tools include Trivy and Docker Scout:

# Scanning a built container image with Trivy in CI pipeline
$ trivy image     --severity HIGH,CRITICAL     --exit-code 1     --ignore-unfixed     devopstribune/core-api:1.4.2

If Trivy detects unpatched Critical vulnerabilities in an upstream library, it exits with return code 1, automatically failing the CI/CD pull request.


11. Production Troubleshooting Runbook for Docker Containers

When operating Docker in production environments, use the following operational triage matrix to rapidly diagnose container failures:

Observed Symptom Root Cause Immediate SRE Remediation
Exit Code 137 Container killed by host Linux kernel OOM killer (exceeded memory limit). Inspect docker inspect <id> --format '{{.State.OOMKilled}}'. Increase container memory limit or tune JVM/Node memory heap flags.
Exit Code 127 Entrypoint binary or dynamic shared library (glibc vs musl) missing inside image. Verify executable path. If using Alpine/Distroless with Go, ensure binary was compiled with CGO_ENABLED=0.
no space left on device Dangling images, build caches, or un-rotated container stdout/stderr log files filling host disk. Execute docker system prune -af --volumes. Configure log rotation in /etc/docker/daemon.json with max-size: 50m.
Container DNS resolution fails Container running on default bridge where embedded DNS is unavailable, or host firewall blocking port 53. Attach container to a user-defined bridge network (docker network connect). Verify host iptables forwarding rules.

12. Frequently Asked Questions

What is the difference between Docker and containerd?

Docker is an end-user container platform encompassing a developer CLI, build engine (BuildKit), volume management, and networking orchestration. containerd is a lightweight, lower-level core container runtime originally built by Docker and donated to the CNCF. Docker Engine internally utilizes containerd to manage the container lifecycle. In Kubernetes environments, Kubernetes communicates directly with containerd via the Container Runtime Interface (CRI), bypassing Docker entirely.

Why should I avoid running containers as root?

By default, root (UID 0) inside a container shares the exact same user ID as root on the host Linux kernel. If an attacker escapes the container through a volume mount misconfiguration or kernel vulnerability, they possess immediate root privileges on the underlying host. Running as an unprivileged user (e.g. UID 10001) prevents host takeover even if the application process is fully compromised.

What is the advantage of distroless images over Alpine?

While Alpine Linux is exceptionally small (~7 MB), it still includes the BusyBox shell, an operating system package manager (apk), and the musl C standard library. Google Distroless images contain strictly your compiled application binary and its direct runtime dependencies (such as CA certificates and glibc). There is no shell, no curl, and no package manager, making it virtually impossible for attackers to run interactive reconnaissance commands.

Related Container & Cloud Infrastructure Guides