Server Monitoring Inside a Docker Container Is Weird
Learn how server monitoring inside a Docker container works. We will give an example of how to use Node Exporter inside a Docker container to analyze a server's storage, CPU, and memory.
Table of Contents 📖
System Monitoring in a Container
Monitoring hardware and operating system level metrics of a server is a little different when the software doing the monitoring is running inside a Docker container. Node Exporter is a commonly used tool to analyze system metrics that is often deployed in a Docker container. However, as Node Exporter is designed to monitor the host system, deploying it in a container requires extra steps to avoid monitoring the container and not the host.
WARNING: Don't forget that Node Exporter exposes the metrics (default port 9100 on /metrics route) but it is Prometheus that scrapes the metrics.
If you want to use a Dockerized Node Exporter to monitor a host, you need to mount the appropriate host directories into the container while excluding any container specific directories. This will tell Node Exporter where to find the host's system information while also ignoring the container's details.
System Monitoring in a Container
The way Node Exporter gathers system information is through collectors. Collectors are responsible for gathering specific information about a system. For example, some commonly used collectors are those below:
- cpu - Gathers CPU metrics.
- meminfo - Gathers memory usage information.
- filesystem - Gathers data on filesystem usage.
- processes - Gathers information about running processes.
Collectors are enabled through the --collector.name flag and disabled by using --no-collector.name. For example, below we enable the processes collector.
node-exporter:
pull_policy: always
image: prom/node-exporter:latest
container_name: ${NODE_EXPORTER_CONTAINER}
restart: always
env_file: .prod.env
volumes:
- /proc:/host/proc:ro
- /sys:/host/sys:ro
- /:/rootfs:ro
expose:
- ${NODE_EXPORTER_PORT}
command:
- --path.procfs=/host/proc
- --path.rootfs=/rootfs
- --path.sysfs=/host/sys
- --collector.processes
INFO: The metrics that Node Exporter exposes are prefixed with node_.
The Dockerized Node Exporter instance makes this even easier for us though as it has some default collector settings already configured. Check out the standard output below from a Node Exporter container. An important line is the --collector.filesystem flags which are used to exclude the container specific directories.
ts=2024-12-28T14:53:32.062Z caller=node_exporter.go:193 level=info msg="Starting node_exporter" version="(version=1.8.2, branch=HEAD, revision=f1e0e8360aa60b6cb5e5cc1560bed348fc2c1895)"
ts=2024-12-28T14:53:32.062Z caller=node_exporter.go:194 level=info msg="Build context" build_context="(go=go1.22.5, platform=linux/amd64, user=root@03d440803209, date=20240714-11:53:45, tags=unknown)"
ts=2024-12-28T14:53:32.066Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(z?ram|loop|fd|(h|s|v|xv)d[a-z]|nvmed+nd+p)d+$
ts=2024-12-28T14:53:32.066Z caller=diskstats_linux.go:265 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data
ts=2024-12-28T14:53:32.066Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/)
ts=2024-12-28T14:53:32.066Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
ts=2024-12-28T14:53:32.066Z caller=node_exporter.go:111 level=info msg="Enabled collectors"
ts=2024-12-28T14:53:32.066Z caller=node_exporter.go:118 level=info collector=arp
ts=2024-12-28T14:53:32.066Z caller=node_exporter.go:118 level=info collector=bcache
ts=2024-12-28T14:53:32.066Z caller=node_exporter.go:118 level=info collector=bonding
ts=2024-12-28T14:53:32.066Z caller=node_exporter.go:118 level=info collector=btrfs
ts=2024-12-28T14:53:32.066Z caller=node_exporter.go:118 level=info collector=conntrack
ts=2024-12-28T14:53:32.066Z caller=node_exporter.go:118 level=info collector=cpu
ts=2024-12-28T14:53:32.066Z caller=node_exporter.go:118 level=info collector=cpufreq
ts=2024-12-28T14:53:32.066Z caller=node_exporter.go:118 level=info collector=diskstats
ts=2024-12-28T14:53:32.066Z caller=node_exporter.go:118 level=info collector=dmi
ts=2024-12-28T14:53:32.066Z caller=node_exporter.go:118 level=info collector=edac
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=entropy
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=fibrechannel
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=filefd
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=filesystem
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=hwmon
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=infiniband
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=ipvs
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=loadavg
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=mdadm
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=meminfo
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=netclass
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=netdev
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=netstat
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=nfs
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=nfsd
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=nvme
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=os
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=powersupplyclass
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=pressure
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=rapl
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=schedstat
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=selinux
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=sockstat
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=softnet
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=stat
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=tapestats
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=textfile
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=thermal_zone
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=time
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=timex
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=udp_queues
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=uname
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=vmstat
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=watchdog
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=xfs
ts=2024-12-28T14:53:32.067Z caller=node_exporter.go:118 level=info collector=zfs
ts=2024-12-28T14:53:32.068Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9100
ts=2024-12-28T14:53:32.068Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9100
Because of these settings, along with the volumes we have mapped into the container, we will analyze our system while ignoring the container that Node Exporter is running in.