Overlay Networks & Docker Swarm

Overlay networks stretch a virtual Layer 2 network across multiple Docker hosts, enabling containers on different machines to communicate as if they were on the same LAN.

How Overlay Works

Docker uses VXLAN (Virtual Extensible LAN) to encapsulate container traffic in UDP packets:

Host A                              Host B
┌─────────────────┐                ┌─────────────────┐
│ Container 1     │                │ Container 2     │
│ 10.0.0.2        │                │ 10.0.0.3        │
└──────┬──────────┘                └──────┬──────────┘
       │ veth                             │ veth
┌──────▼──────────┐                ┌─────▼───────────┐
│ br0 (overlay)   │  VXLAN/UDP     │ br0 (overlay)   │
│   10.0.0.0/24   ├────────────────┤   10.0.0.0/24   │
└─────────────────┘  port 4789     └─────────────────┘

The VXLAN VNI (Virtual Network Identifier) isolates different overlay networks.

Docker Swarm

Swarm is Docker's built-in cluster manager. It adds:

• Manager nodes: schedule tasks, maintain cluster state (Raft consensus)
• Worker nodes: run tasks (containers)

# Initialize a swarm on the first manager
docker swarm init --advertise-addr <manager-ip>

# Join workers (token from init output)
docker swarm join --token <worker-token> <manager-ip>:2377

# Join additional managers
docker swarm join --token <manager-token> <manager-ip>:2377

Overlay Networks in Swarm

# Create a multi-host overlay network
docker network create \
  --driver overlay \
  --subnet 10.0.9.0/24 \
  my-overlay

# All swarm services on this network can communicate

The ingress network (created automatically) carries external traffic into Swarm services.

Swarm Services

A service is the Swarm equivalent of a docker run:

# Replicated service (N copies spread across cluster)
docker service create \
  --name web \
  --replicas 3 \
  --network my-overlay \
  --publish published=80,target=80 \
  nginx

# Global service (one copy per node)
docker service create \
  --name monitor \
  --mode global \
  prom/node-exporter

# Scale
docker service scale web=5

# Rolling update
docker service update \
  --image nginx:1.26 \
  --update-parallelism 1 \
  --update-delay 10s \
  web

Service Discovery in Swarm

Swarm's DNS resolves:

• Service name → Virtual IP (VIP) of the service
• Swarm load-balances connections across replicas via IPVS

# VIP-based load balancing (default)
docker service create --name api --endpoint-mode vip ...

# DNS round-robin (each DNS lookup returns a different replica IP)
docker service create --name api --endpoint-mode dnsrr ...

Ingress Load Balancing (Routing Mesh)

Any node in the swarm can receive traffic for any published port, even if no replica runs on that node:

External client → any node:80
                → routing mesh
                → replica (wherever it lives in the cluster)

This is implemented with iptables + IPVS.

Overlay Encryption

docker network create \
  --driver overlay \
  --opt encrypted \
  secure-overlay

Encrypts the VXLAN data plane using AES-GCM (256-bit). Adds CPU overhead.

Summary

• Overlay networks use VXLAN to extend L2 across multiple hosts
• Docker Swarm orchestrates multi-node deployments with managers and workers
• Service VIPs + IPVS provide built-in load balancing
• The routing mesh lets any node receive traffic for any published port
• Overlay encryption adds AES-GCM data plane security