容器监控原理发现

docker采用cgroup进行资源控制，在本地计算机上找到cgroup的目录（不同操作系统不同），例如:

/sys/fs/cgroup
.
├── blkio
├── cpu -> cpu,cpuacct
├── cpuacct -> cpu,cpuacct
├── cpu,cpuacct
├── cpuset
├── devices
├── freezer
├── hugetlb
├── memory
├── net_cls -> net_cls,net_prio
├── net_cls,net_prio
├── net_prio -> net_cls,net_prio
├── perf_event
├── pids
├── systemd
└── unified

此处以内存为例，进入memory目录下, 可以发现存在一个以docker命名的目录

├── cgroup.clone_children
├── cgroup.event_control
├── cgroup.procs
├── cgroup.sane_behavior
├── docker
├── init.scope
├── memory.failcnt
├── memory.force_empty
├── memory.kmem.failcnt
├── memory.kmem.limit_in_bytes
├── memory.kmem.max_usage_in_bytes
├── memory.kmem.slabinfo
├── memory.kmem.tcp.failcnt
├── memory.kmem.tcp.limit_in_bytes
├── memory.kmem.tcp.max_usage_in_bytes
├── memory.kmem.tcp.usage_in_bytes
├── memory.kmem.usage_in_bytes
├── memory.limit_in_bytes
├── memory.max_usage_in_bytes
├── memory.memsw.failcnt
├── memory.memsw.limit_in_bytes
├── memory.memsw.max_usage_in_bytes
├── memory.memsw.usage_in_bytes
├── memory.move_charge_at_immigrate
├── memory.numa_stat
├── memory.oom_control
├── memory.pressure_level
├── memory.soft_limit_in_bytes
├── memory.stat
├── memory.swappiness
├── memory.usage_in_bytes
├── memory.use_hierarchy
├── notify_on_release
├── release_agent
├── system.slice
├── tasks
├── user.slice

进入docker目录

.
├── 5fda52353901c68db2981102d8f0c2d3b0111c773ec17dc11f5b33bf047cab20
├── 83c0e6c8ed5b0f47975bc7f1728c8c1bb03a3373d48bef2aab95006fcdecbc20
├── cgroup.clone_children
├── cgroup.event_control
├── cgroup.procs
├── memory.failcnt
├── memory.force_empty
├── memory.kmem.failcnt
├── memory.kmem.limit_in_bytes
├── memory.kmem.max_usage_in_bytes
├── memory.kmem.slabinfo
├── memory.kmem.tcp.failcnt
├── memory.kmem.tcp.limit_in_bytes
├── memory.kmem.tcp.max_usage_in_bytes
├── memory.kmem.tcp.usage_in_bytes
├── memory.kmem.usage_in_bytes
├── memory.limit_in_bytes
├── memory.max_usage_in_bytes
├── memory.memsw.failcnt
├── memory.memsw.limit_in_bytes
├── memory.memsw.max_usage_in_bytes
├── memory.memsw.usage_in_bytes
├── memory.move_charge_at_immigrate
├── memory.numa_stat
├── memory.oom_control
├── memory.pressure_level
├── memory.soft_limit_in_bytes
├── memory.stat
├── memory.swappiness
├── memory.usage_in_bytes
├── memory.use_hierarchy
├── notify_on_release
└── tasks

2 directories, 31 files
➜  docker docker ps -a
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                    NAMES
5fda52353901        google/cadvisor:latest   "/usr/bin/cadvisor -…"   2 hours ago         Up 2 hours         0.0.0.0:8080->8080/tcp   cadvisor
83c0e6c8ed5b        centos                   "/bin/bash"              2 hours ago         Up 2 hours                        zwf

发现有两个目录与目前启动的容器ID重名，进入看下

➜  5fda52353901c68db2981102d8f0c2d3b0111c773ec17dc11f5b33bf047cab20 tree -L 1
.
├── cgroup.clone_children
├── cgroup.event_control
├── cgroup.procs
├── memory.failcnt
├── memory.force_empty
├── memory.kmem.failcnt
├── memory.kmem.limit_in_bytes
├── memory.kmem.max_usage_in_bytes
├── memory.kmem.slabinfo
├── memory.kmem.tcp.failcnt
├── memory.kmem.tcp.limit_in_bytes
├── memory.kmem.tcp.max_usage_in_bytes
├── memory.kmem.tcp.usage_in_bytes
├── memory.kmem.usage_in_bytes
├── memory.limit_in_bytes
├── memory.max_usage_in_bytes
├── memory.memsw.failcnt
├── memory.memsw.limit_in_bytes
├── memory.memsw.max_usage_in_bytes
├── memory.memsw.usage_in_bytes
├── memory.move_charge_at_immigrate
├── memory.numa_stat
├── memory.oom_control
├── memory.pressure_level
├── memory.soft_limit_in_bytes
├── memory.stat
├── memory.swappiness
├── memory.usage_in_bytes
├── memory.use_hierarchy
├── notify_on_release
└── tasks

这个里面就存放的是启动的docker容器当前内存资源的统计与限制，其中tasks存放的是容器中运行的收到限制的进程的PID，所以如果要做监控的话，只需要读取cgroup中的对应文件即可。
CAdvisor在做的时候需要用户手动将本机的cgroup目录挂载到容器内部，这样屏蔽了因操作系统导致cgroup路径差异的问题，然后的处理方式就是读取cgroup中的统计数据，然后在内存中汇总上报。