Abberation - missing cadvisor metrics

[srajappa]

K3s version: k3s version v1.31.5+k3s1 (56ec5dd4)

I am seeing an issue where cadvisor metrics are missing. The weird issue is, it shows up in one k3s cluster but missing in another.

For example, we are able to view the metrics for pods running on this cluster called dd-uat-b

> kubectl get nodes
> kubectl get nodes -o wide
NAME                                               STATUS   ROLES                  AGE     VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
<node1>    Ready    <none>                 3d11h   v1.31.5+k3s1   192.160.9.141   <none>        Ubuntu 22.04.3 LTS   5.10.0-35-amd64   containerd://1.7.23-k3s2
<node2>    Ready    <none>                 4d17h   v1.31.5+k3s1   192.160.9.132   <none>        Ubuntu 22.04.3 LTS   5.10.0-35-amd64   containerd://1.7.23-k3s2
<node3>    Ready    <none>                 19d     v1.31.5+k3s1   192.160.1.48    <none>        Ubuntu 22.04.3 LTS   5.10.0-35-amd64   containerd://1.7.23-k3s2
<server1>           Ready    control-plane,master   101d    v1.31.5+k3s1   192.160.0.159   <none>        Ubuntu 22.04.3 LTS   5.10.0-35-amd64   containerd://1.7.23-k3s2
> kubectl get --raw /api/v1/nodes/<node1>/proxy/metrics/cadvisor | grep 'container_cpu_cfs_periods_total'
# HELP container_cpu_cfs_periods_total Number of elapsed enforcement period intervals.
# TYPE container_cpu_cfs_periods_total counter
container_cpu_cfs_periods_total{container="",id="/",image="",name="",namespace="",pod=""} ...
container_cpu_cfs_periods_total{container="",id="/kubepods/burstable/pod...",image="",name="",namespace="...",pod="..."} ...
container_cpu_cfs_periods_total{container="...",id="/kubepods/burstable/pod.../...",image="...",name="...",namespace="...",pod="..."} ...
# ...more lines...

But dd-b which has a similar cluster-configuration doesn't show up ALL metrics.

> kubectl get nodes -o wide
NAME                                           STATUS   ROLES                  AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
<node1>      Ready    <none>                 8d    v1.31.5+k3s1   192.160.1.65    <none>        Ubuntu 22.04.3 LTS   5.10.0-35-amd64   containerd://1.7.23-k3s2
<node2>      Ready    <none>                 8d    v1.31.5+k3s1   192.160.1.6     <none>        Ubuntu 22.04.3 LTS   5.10.0-35-amd64   containerd://1.7.23-k3s2
<node3>   Ready    <none>                 8d    v1.31.5+k3s1   192.160.1.5     <none>        Ubuntu 22.04.3 LTS   5.10.0-35-amd64   containerd://1.7.23-k3s2
<server1>           Ready    control-plane,master   23d   v1.31.5+k3s1   192.160.0.118   <none>        Ubuntu 22.04.3 LTS   5.10.0-35-amd64   containerd://1.7.23-k3s2
> kubectl get --raw /api/v1/nodes/<nodeA>/proxy/metrics/cadvisor | grep 'container_cpu_cfs_periods_total'
# HELP container_cpu_cfs_periods_total Number of elapsed enforcement period intervals.
# TYPE container_cpu_cfs_periods_total counter
container_cpu_cfs_periods_total{container="",id="/",image="",name="",namespace="",pod=""} ...
container_cpu_cfs_periods_total{container="",id="/kubepods/burstable/pod...",image="",name="",namespace="...",pod="..."} ...
container_cpu_cfs_periods_total{container="...",id="/kubepods/burstable/pod.../...",image="...",name="...",namespace="...",pod="..."}
<NOT MANY LINES>

I think this problem maybe a TANGENT to this issue talked here.

I wanted to know if there is someway to understand how cadvisor may have a missing configuration in the latter k3s cluster ? Also wanted see if I can find cadvisor error logs, currently I have no access logs for incoming API calls on the k3s agent service.

Please advise.

[brandond]

Are you using the same container runtime on both clusters? You didn't show the -o wide output of the node list so I can't tell.

[srajappa]

Apologies! While masking sensitive information I didn't pay much attention to the -o wide info. Here is the information below, will also edit the discussion content soon.

dd-uat-b cluster where cadvisor metrics are all visible.

> kubectl get nodes -o wide
NAME                                               STATUS   ROLES                  AGE     VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
<node1>    Ready    <none>                 3d11h   v1.31.5+k3s1   192.160.9.141   <none>        Ubuntu 22.04.3 LTS   5.10.0-35-amd64   containerd://1.7.23-k3s2
<node2>    Ready    <none>                 4d17h   v1.31.5+k3s1   192.160.9.132   <none>        Ubuntu 22.04.3 LTS   5.10.0-35-amd64   containerd://1.7.23-k3s2
<node3>    Ready    <none>                 19d     v1.31.5+k3s1   192.160.1.48    <none>        Ubuntu 22.04.3 LTS   5.10.0-35-amd64   containerd://1.7.23-k3s2
<server1>           Ready    control-plane,master   101d    v1.31.5+k3s1   192.160.0.159   <none>        Ubuntu 22.04.3 LTS   5.10.0-35-amd64   containerd://1.7.23-k3s2

dd-b cluster where cadvisor metrics are all NOT visible.

> kubectl get nodes -o wide
NAME                                           STATUS   ROLES                  AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
<node1>      Ready    <none>                 8d    v1.31.5+k3s1   192.160.1.65    <none>        Ubuntu 22.04.3 LTS   5.10.0-35-amd64   containerd://1.7.23-k3s2
<node2>      Ready    <none>                 8d    v1.31.5+k3s1   192.160.1.6     <none>        Ubuntu 22.04.3 LTS   5.10.0-35-amd64   containerd://1.7.23-k3s2
<node3>   Ready    <none>                 8d    v1.31.5+k3s1   192.160.1.5     <none>        Ubuntu 22.04.3 LTS   5.10.0-35-amd64   containerd://1.7.23-k3s2
<server1>           Ready    control-plane,master   23d   v1.31.5+k3s1   192.160.0.118   <none>        Ubuntu 22.04.3 LTS   5.10.0-35-amd64   containerd://1.7.23-k3s2

[brandond]

Yeah I'm not sure what to suggest. The cadvisor stuff is all built into the kubelet so there's not really anything to configure.

Do you maybe have cgroupv1 or hybrid cgroups on some nodes, and cgroup v2 on others? Note that some metrics may not exist depending on kernel cgroup versions.

[srajappa]

The cadvisor stuff is all built into the kubelet so there's not really anything to configure.

Ah! that's going to be tough, it would be very useful to see any information to why it's not able to gather some metrics.

Do you maybe have cgroupv1 or hybrid cgroups on some nodes

That's a great question. I checked, both these clusters' nodes have cgroupsv2 and not hybrid.

Note that some metrics may not exist depending on kernel cgroup versions.

That's something worth exploring and see if there are some metrics not getting exposed because the kernel.

Ah!! even the kernel version is the same.

[srajappa]

Just found out why these metrics didn't come up on the other cluster.

No CPU Limits were placed on the POD OBJECTS for the ones running in the other cluster.

It turns out without CPU Limits the metrics measuring the CPU throttling is dropped.

google/cadvisor#3154

Thank you for help :)