Spegel functioning but no cache hits

[craigcabrey]

k3s: v1.29.5+k3s1

I used this trick from another answer to look at spegel stats:

kubectl get --raw /api/v1/nodes/<node>/proxy/metrics | grep -F 'spegel'

With sample output:

libp2p_rcmgr_streams{dir="inbound",protocol="/spegel/kad/1.0.0",scope="protocol"} 0
libp2p_rcmgr_streams{dir="outbound",protocol="/spegel/kad/1.0.0",scope="protocol"} 1
# HELP spegel_advertised_image_digests Number of image digests advertised to be available.
# TYPE spegel_advertised_image_digests gauge
spegel_advertised_image_digests{registry="docker.io"} 35
spegel_advertised_image_digests{registry="gcr.io"} 1
spegel_advertised_image_digests{registry="ghcr.io"} 18
spegel_advertised_image_digests{registry="quay.io"} 15
spegel_advertised_image_digests{registry="registry-1.docker.io"} 1
spegel_advertised_image_digests{registry="registry.k8s.io"} 5
# HELP spegel_advertised_image_tags Number of image tags advertised to be available.
# TYPE spegel_advertised_image_tags gauge
spegel_advertised_image_tags{registry="docker.io"} 14
spegel_advertised_image_tags{registry="gcr.io"} 1
spegel_advertised_image_tags{registry="ghcr.io"} 3
spegel_advertised_image_tags{registry="quay.io"} 3
spegel_advertised_image_tags{registry="registry.k8s.io"} 1
# HELP spegel_advertised_images Number of images advertised to be available.
# TYPE spegel_advertised_images gauge
spegel_advertised_images{registry="docker.io"} 49
spegel_advertised_images{registry="gcr.io"} 2
spegel_advertised_images{registry="ghcr.io"} 21
spegel_advertised_images{registry="quay.io"} 18
spegel_advertised_images{registry="registry-1.docker.io"} 1
spegel_advertised_images{registry="registry.k8s.io"} 6
# HELP spegel_advertised_keys Number of keys advertised to be available.
# TYPE spegel_advertised_keys gauge
spegel_advertised_keys{registry="docker.io"} 355
spegel_advertised_keys{registry="gcr.io"} 14
spegel_advertised_keys{registry="ghcr.io"} 287
spegel_advertised_keys{registry="quay.io"} 169
spegel_advertised_keys{registry="registry-1.docker.io"} 4
spegel_advertised_keys{registry="registry.k8s.io"} 49

I have the required registries.yaml deployed on every node. Some of my images use latest, but many don't. Furthermore, I use k8s-digester (as recommended by the spegel project itself) to resolve image tags to a digest. This should make for a very cache friendly workload.

Yet, I see no cache hits. My only suspicion is that it's related to my dual stack setup, with IPv6 as a priority (i.e. the first address). Is there some way to access the logs of spegel?

[brandond]

Spegel logs are mixed in with all the rest of the k3s logs in the systemd journal. If you run with --debug you'll get more of them. You can also check the containerd logs.

If you're using ipv6 as the primary address family on your nodes, you're probably running into #9897 (comment)

[craigcabrey]

Finally getting back to this. I don't even see the configs referenced in that issue being generated for me. For example, here is a worker node:

root@ms01-node-1:/var/lib/rancher/k3s/agent/etc/containerd# ls -l certs.d/
total 0
drwx------. 2 root root 24 Aug 23 17:42 '[::1]:6443'
drwx------. 2 root root 24 Aug 23 17:42  _default
root@ms01-node-1:/var/lib/rancher/k3s/agent# journalctl -u k3s-agent | grep toml
[snip]
Aug 16 07:12:41 ms01-node-1 k3s[3064083]: time="2024-08-16T07:12:41Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.tom -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"
Aug 23 17:42:35 ms01-node-1 k3s[1814567]: time="2024-08-23T17:42:35Z" level=info msg="Running containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.tom -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd"

All of my control plane nodes have the embedded-registry: true field set.

[brandond]

Yes, the issue is going to be related to the IPV6 problem I linked above.

[craigcabrey]

Checking back in on this:

#9897 (comment)

At this point I am going to close this with a suggestion that anyone who needs this to work use K3s 1.31+ with containerd 2.0.

I am running v1.31.12+k3s1 on all nodes, but Spegel still doesn't seem to function as expected. Spegel is running, and I can query the counters:

craigcabrey@miniboi ~ % kubectl get --raw /api/v1/nodes/ms01-node-1/proxy/metrics | grep -F 'spegel'
libp2p_rcmgr_streams{dir="inbound",protocol="/spegel/kad/1.0.0",scope="protocol"} 0
libp2p_rcmgr_streams{dir="outbound",protocol="/spegel/kad/1.0.0",scope="protocol"} 0
# HELP spegel_advertised_image_digests Number of image digests advertised to be available.
# TYPE spegel_advertised_image_digests gauge
spegel_advertised_image_digests{registry="docker-registry3.mariadb.com"} 1
spegel_advertised_image_digests{registry="docker.gitea.com"} 1
spegel_advertised_image_digests{registry="docker.io"} 91
spegel_advertised_image_digests{registry="ghcr.io"} 44
spegel_advertised_image_digests{registry="git.craigcabrey.com"} 2
spegel_advertised_image_digests{registry="public.ecr.aws"} 1
spegel_advertised_image_digests{registry="quay.io"} 35
spegel_advertised_image_digests{registry="registry.k8s.io"} 22
# HELP spegel_advertised_image_tags Number of image tags advertised to be available.
# TYPE spegel_advertised_image_tags gauge
spegel_advertised_image_tags{registry="docker-registry3.mariadb.com"} 1
spegel_advertised_image_tags{registry="docker.io"} 31
spegel_advertised_image_tags{registry="ghcr.io"} 13
spegel_advertised_image_tags{registry="quay.io"} 14
spegel_advertised_image_tags{registry="registry.k8s.io"} 5
# HELP spegel_advertised_images Number of images advertised to be available.
# TYPE spegel_advertised_images gauge
spegel_advertised_images{registry="docker-registry3.mariadb.com"} 2
spegel_advertised_images{registry="docker.gitea.com"} 1
spegel_advertised_images{registry="docker.io"} 122
spegel_advertised_images{registry="ghcr.io"} 57
spegel_advertised_images{registry="git.craigcabrey.com"} 2
spegel_advertised_images{registry="public.ecr.aws"} 1
spegel_advertised_images{registry="quay.io"} 49
spegel_advertised_images{registry="registry.k8s.io"} 27
# HELP spegel_advertised_keys Number of keys advertised to be available.
# TYPE spegel_advertised_keys gauge
spegel_advertised_keys{registry="docker-registry3.mariadb.com"} 16
spegel_advertised_keys{registry="docker.gitea.com"} 13
spegel_advertised_keys{registry="docker.io"} 851
spegel_advertised_keys{registry="ghcr.io"} 674
spegel_advertised_keys{registry="git.craigcabrey.com"} 33
spegel_advertised_keys{registry="public.ecr.aws"} 11
spegel_advertised_keys{registry="quay.io"} 480
spegel_advertised_keys{registry="registry.k8s.io"} 286

I'm not sure, but it seems like the generated configs aren't correct?

root@ms01-node-1:/var/lib/rancher/k3s/agent/etc/containerd/certs.d# ls -Rl
.:
total 0
drwx------. 2 root root 24 Sep 18 14:51 '[::1]:6443'
drwx------. 2 root root 24 Sep 18 14:51  _default

'./[::1]:6443':
total 4
-rw-r--r--. 1 root root 296 Sep 18 14:51 hosts.toml

./_default:
total 4
-rw-r--r--. 1 root root 348 Sep 18 14:51 hosts.toml
root@ms01-node-1:/var/lib/rancher/k3s/agent/etc/containerd/certs.d# cat \[\:\:1\]\:6443/hosts.toml 
# File generated by k3s. DO NOT EDIT.
server = "https://[::1]:6443/v2"
capabilities = ["pull", "resolve", "push"]
ca = ["/var/lib/rancher/k3s/agent/server-ca.crt"]
client = [["/var/lib/rancher/k3s/agent/client-k3s-controller.crt", "/var/lib/rancher/k3s/agent/client-k3s-controller.key"]]

[host]
root@ms01-node-1:/var/lib/rancher/k3s/agent/etc/containerd/certs.d# cat _default/hosts.toml 
# File generated by k3s. DO NOT EDIT.
server = ""
capabilities = ["pull", "resolve", "push"]

[host]

[host."https://[::1]:6443/v2"]
  capabilities = ["pull", "resolve"]
  ca = ["/var/lib/rancher/k3s/agent/server-ca.crt"]
  client = [["/var/lib/rancher/k3s/agent/client-k3s-controller.crt", "/var/lib/rancher/k3s/agent/client-k3s-controller.key"]]

Would love any ideas whenever someone gets a chance.

[brandond]

The metrics are only updated every 9 minutes. This is tracked upstream: spegel-org/spegel#1023

What do you think looks wrong about the hosts.toml files? They look fine to me...