I am trying to monitoring root partitions inside containers using gcr.io/cadvisor/cadvisor:v0.37.0 and Prometheus.
Docker is set up with:
"storage-driver": "overlay2",
"storage-opts": ["overlay2.size=5G"],
For example, for the following container, I would like to get the 5% (actually 4.06% when computed exactly)
[root@infra-05 docker-compose]# docker exec -ti infra-05_logging-elasticsearch df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 5242880 213172 5029708 5% /
tmpfs 65536 0 65536 0% /dev
tmpfs 8023452 0 8023452 0% /sys/fs/cgroup
/dev/mapper/vg0-varlibdockerlv 41932800 30960284 10972516 74% /etc/hosts
shm 65536 0 65536 0% /dev/shm
/dev/mapper/vg0-vardockerelasticsearchlv 146785280 24772532 122012748 17% /usr/share/elasticsearch/data
/dev/mapper/vg0-lv_root 3997376 3023344 747936 81% /usr/share/elasticsearch/config/log4j2.properties
tmpfs 8023452 0 8023452 0% /proc/acpi
tmpfs 8023452 0 8023452 0% /proc/scsi
tmpfs 8023452 0 8023452 0% /sys/firmware
Following query in Prometheus:
container_fs_usage_bytes{instance="infra-05.node.consul:9103",name=".elasticsearch."}/container_fs_limit_bytes{instance="infra-05.node.consul:9103",name=".elasticsearch."}*100
brings back a single line:
{container_label_co_elastic_logs_module="elasticsearch",container_label_com_docker_compose_config_hash="ad29cbce151d36e15c4b13d72b1818bc0808e1936890e51e46e9c6e0816bb6a7",container_label_com_docker_compose_container_number="1",container_label_com_docker_compose_oneoff="False",container_label_com_docker_compose_project="docker-compose",container_label_com_docker_compose_service="logging-elasticsearch",container_label_com_docker_compose_version="1.21.2",container_label_org_label_schema_build_date="2019-11-26T01:06:52.520070Z",container_label_org_label_schema_license="Elastic-License",container_label_org_label_schema_name="Elasticsearch",container_label_org_label_schema_schema_version="1.0",container_label_org_label_schema_url="https://www.elastic.co/products/elasticsearch",container_label_org_label_schema_usage="https://www.elastic.co/guide/en/elasticsearch/reference/index.html",container_label_org_label_schema_vcs_ref="e9ccaed468e2fac2275a3761849cbee64b39519f",container_label_org_label_schema_vcs_url="https://github.com/elastic/elasticsearch",container_label_org_label_schema_vendor="Elastic",container_label_org_label_schema_version="7.5.0",container_label_org_opencontainers_image_created="2019-11-26T01:06:52.520070Z",container_label_org_opencontainers_image_documentation="https://www.elastic.co/guide/en/elasticsearch/reference/index.html",container_label_org_opencontainers_image_licenses="Elastic-License",container_label_org_opencontainers_image_revision="e9ccaed468e2fac2275a3761849cbee64b39519f",container_label_org_opencontainers_image_source="https://github.com/elastic/elasticsearch",container_label_org_opencontainers_image_title="Elasticsearch",container_label_org_opencontainers_image_url="https://www.elastic.co/products/elasticsearch",container_label_org_opencontainers_image_vendor="Elastic",container_label_org_opencontainers_image_version="7.5.0",customer="Xenit",device="/dev/mapper/vg0-varlibdockerlv",environment="Xenit_PROD",id="/docker/677f57566219f459f30d6c4abac0235b3ab43c0f8031747bf676dfa9b33d1047",image="elasticsearch:7.5.0",instance="infra-05.node.consul:9103",job="cadvisor",name="infra-05_logging-elasticsearch",service="cadvisor"} | 0.5084897741147741
I noticed that the root partition in that container is available if I query for device=~overlay.*:
container_fs_usage_bytes{instance="infra-05.node.consul:9103",device="overlay.*"}/container_fs_limit_bytes{instance="infra-05.node.consul:9103",device="overlay.*"}*100
{customer="Xenit",device="overlay_0-109",environment="Xenit_PROD",id="/",instance="infra-05.node.consul:9103",job="cadvisor",service="cadvisor"} | 4.0659332275390625
......................
But how to link the device "overlay_0-109" with the container infra-05_logging-elasticsearch?
