Containers internals

Contract

Field	Bound contract
Trigger	A container's isolation or resource limit needs explaining or debugging, a seccomp profile needs writing, a minimal container is being built without a daemon, or an escape vector needs assessing.
Authority	Reversible local. The write set is namespaces and cgroups created for the session under /sys/fs/cgroup, overlay mounts on user-named directories, an OCI bundle directory, and seccomp filters applied to processes the skill starts. Rollback is stopping those processes, unmounting, and removing the cgroup directory. Most steps need root or a user namespace. No remote mutation.
Side effect	Kernel namespaces, cgroups, and mounts exist while the experiment runs.
Done	The isolation or limit in question is reproduced with the raw primitive, the observed behavior matches the explanation, and every created object has its teardown recorded.

Inputs

• Question or symptom (required): an OOM kill, CPU throttling, a permission denial inside the container, a mount failure, or a concept.
• Target (optional): a running container's PID, or a rootfs directory for a manual container.
• Privilege (required to know): root, or an unprivileged user relying on user namespaces. Several steps differ.

Procedure

1. Read and enter namespaces. Each namespace type isolates one resource. Done when: the target process's namespace inodes are listed and, when needed, a shell runs inside them.

ls -la /proc/self/ns/            # one link per namespace type
readlink /proc/1234/ns/net       # compare inodes to see who shares a namespace
nsenter -t <pid> -m -u -i -n -p bash
unshare --fork --mount-proc --pid --net --uts --ipc bash   # a manual container shell

Flag	Isolates
CLONE_NEWNS	Mount table
CLONE_NEWPID	Process ids
CLONE_NEWNET	Network stack
CLONE_NEWUTS	Hostname and domain name
CLONE_NEWIPC	System V IPC and POSIX message queues
CLONE_NEWUSER	Uid and gid mappings
CLONE_NEWCGROUP	The cgroup root the process sees
CLONE_NEWTIME	Boot-time and monotonic clock offsets

#define _GNU_SOURCE
#include <sched.h>
#include <sys/mount.h>
#include <unistd.h>

static int container_init(void *arg)
{
    sethostname("container", 9);
    mount("proc", "/proc", "proc", 0, NULL);
    execv("/bin/sh", (char *[]){"/bin/sh", NULL});
    return 1;
}

static char stack[1024 * 1024];
/* The child stack grows down, so pass the top of the buffer. */
clone(container_init, stack + sizeof stack,
      CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET | SIGCHLD, NULL);

2. Apply a cgroups v2 limit. The unified hierarchy at /sys/fs/cgroup exposes one directory per group; a process joins by writing its pid to cgroup.procs. Done when: the limit file holds the value and memory.events or cpu.stat shows the effect under load.

mkdir /sys/fs/cgroup/mycontainer
echo $$ > /sys/fs/cgroup/mycontainer/cgroup.procs
echo 256M > /sys/fs/cgroup/mycontainer/memory.max     # hard memory limit
echo 50 > /sys/fs/cgroup/mycontainer/cpu.weight        # share relative to siblings; default 100
echo "50000 100000" > /sys/fs/cgroup/mycontainer/cpu.max   # quota and period in microseconds
echo "default 100" > /sys/fs/cgroup/mycontainer/io.weight
cat /proc/self/cgroup
cat /sys/fs/cgroup/mycontainer/memory.events           # oom and oom_kill counters

3. Build the filesystem with overlayfs. Reads fall through to the lower layers; writes land in the upper directory; workdir is overlayfs bookkeeping and must be empty and on the same filesystem as upperdir. Done when: the merged mount shows the union and a write appears only in upperdir.

mount -t overlay overlay -o lowerdir=lower1:lower2,upperdir=upper,workdir=work merged

Docker's overlay2 driver keeps its layers under /var/lib/docker/overlay2.

4. Run the bundle with runc. runc spec writes a config.json whose ociVersion matches the installed runtime (runc 1.5.1 writes 1.3.0); do not hand-edit that field. Edit process.args, linux.namespaces, linux.resources, process.capabilities, and linux.seccomp. Done when: runc run starts the container and runc list shows it.

mkdir -p mycontainer/rootfs        # populate rootfs first
runc spec -b mycontainer           # add --rootless when not root
runc run -b mycontainer mycontainer
runc list

5. Filter syscalls with seccomp-bpf. Done when: the filter loads and the blocked syscall returns the chosen action.

#include <seccomp.h>
#include <errno.h>

scmp_filter_ctx ctx = seccomp_init(SCMP_ACT_ALLOW);
seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(mount), 0);
seccomp_rule_add(ctx, SCMP_ACT_ERRNO(EPERM), SCMP_SYS(pivot_root), 0);
seccomp_load(ctx);

Action	Effect
SCMP_ACT_KILL_PROCESS	Kill the whole process (SCMP_ACT_KILL kills only the calling thread)
SCMP_ACT_ERRNO(n)	Fail the call with errno n
SCMP_ACT_TRAP	Deliver SIGSYS
SCMP_ACT_TRACE	Notify a ptrace tracer
SCMP_ACT_LOG	Allow and write an audit record
SCMP_ACT_NOTIFY	Hand the call to a user-space supervisor
SCMP_ACT_ALLOW	Permit

Docker's default profile ships in the moby repository under profiles/seccomp/default.json. To find the syscall a profile is missing, run the workload under strace -f first.

6. Drop capabilities. A container process should run as non-root with the smallest bounding set that still works. Done when: capsh --print or getcap shows only the intended capabilities.

capsh --drop=all --caps="cap_net_bind_service+eip" -- -c '/app/server'
setcap cap_net_bind_service+ep /usr/bin/myserver
getcap /usr/bin/myserver

7. Explain rootless mode. A user namespace maps container uid 0 to an unprivileged host uid, so root inside is an ordinary user outside. Rootless containers cannot mount most filesystems and hold no CAP_SYS_ADMIN on the host. Done when: /proc/<pid>/uid_map for the container shows the mapping.

cat /proc/self/uid_map      # "0 1000 1" maps container uid 0 to host uid 1000

8. Stack the escape mitigations: user namespace, seccomp, a mandatory access control profile (AppArmor or SELinux), a dropped capability set, a read-only root, no-new-privileges, and Landlock for filesystem scope. Known escape vectors are a mounted container-engine socket, privileged mode, kernel CVEs, and /proc leaks. Done when: each layer in use is named and the missing ones are listed.

docker run --read-only --cap-drop=ALL --security-opt=no-new-privileges \
  --security-opt seccomp=default.json myimage

For policy depth (SELinux, AppArmor, seccomp) use kernel-security. To trace what a container does at the syscall level, use ebpf. For the kernel side of cgroups and namespaces, use kernel-internals.

Failure and recovery

Failure class	Behavior
Container OOM-killed	memory.max was exceeded; memory.events shows oom_kill. Raise the limit or fix the leak.
CPU throttled	cpu.max quota is low; cpu.stat shows nr_throttled. Raise the quota or use cpu.weight for a soft share.
Permission denied inside the container	A needed capability was dropped. Add that one capability, not CAP_SYS_ADMIN.
Killed by seccomp at start	The profile lacks a syscall the runtime needs. Find it with strace -f and allow that syscall only.
Overlay mount fails	workdir is not empty or sits on a different filesystem than upperdir. Clean it and retry.
Rootless mount fails	The user namespace forbids it. Bind-mount from the host instead.

Output

The primitive used for each isolation or limit, the commands run, the observed effect, and the teardown list: cgroup directories to remove, mounts to unmount, and processes to stop.