Clean up old CAs after K3s CA rotation

[crtvmn]

Hello,

I used the following documentation https://docs.k3s.io/cli/certificate#rotating-self-signed-ca-certificates to rotate the self-sifgned CAs in my K3S cluster (with the cross-signed certificate).

These are certificates before rotation:

# /opt/bin/k3s certificate check -o table
INFO[0000] Server detected, checking agent and server certificates 

FILENAME                          SUBJECT                            USAGES                  EXPIRES                  RESIDUAL TIME   STATUS
--------                          -------                            ------                  -------                  -------------   ------
client-k3s-cloud-controller.crt   k3s-cloud-controller-manager       ClientAuth              Oct 10, 2026 12:29 UTC   1 year          OK
client-k3s-cloud-controller.crt   k3s-client-ca@1760099372           CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client-scheduler.crt              system:kube-scheduler              ClientAuth              Oct 10, 2026 12:29 UTC   1 year          OK
client-scheduler.crt              k3s-client-ca@1760099372           CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
kube-scheduler.crt                kube-scheduler                     ServerAuth              Oct 10, 2026 12:29 UTC   1 year          OK
kube-scheduler.crt                k3s-server-ca@1760099372           CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client-auth-proxy.crt             system:auth-proxy                  ClientAuth              Oct 10, 2026 12:29 UTC   1 year          OK
client-auth-proxy.crt             k3s-request-header-ca@1760099372   CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client-controller.crt             system:kube-controller-manager     ClientAuth              Oct 10, 2026 12:29 UTC   1 year          OK
client-controller.crt             k3s-client-ca@1760099372           CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
kube-controller-manager.crt       kube-controller-manager            ServerAuth              Oct 10, 2026 12:29 UTC   1 year          OK
kube-controller-manager.crt       k3s-server-ca@1760099372           CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client.crt                        etcd-client                        ClientAuth              Oct 10, 2026 12:29 UTC   1 year          OK
client.crt                        etcd-server-ca@1760099372          CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
server-client.crt                 etcd-server                        ServerAuth,ClientAuth   Oct 10, 2026 12:29 UTC   1 year          OK
server-client.crt                 etcd-server-ca@1760099372          CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
peer-server-client.crt            etcd-peer                          ServerAuth,ClientAuth   Oct 10, 2026 12:29 UTC   1 year          OK
peer-server-client.crt            etcd-peer-ca@1760099372            CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client-supervisor.crt             system:k3s-supervisor              ClientAuth              Oct 10, 2026 12:29 UTC   1 year          OK
client-supervisor.crt             k3s-client-ca@1760099372           CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client-kube-proxy.crt             system:kube-proxy                  ClientAuth              Oct 10, 2026 12:30 UTC   1 year          OK
client-kube-proxy.crt             k3s-client-ca@1760099372           CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client-kubelet.crt                system:node:controller0            ClientAuth              Oct 10, 2026 12:30 UTC   1 year          OK
client-kubelet.crt                k3s-client-ca@1760099372           CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
serving-kubelet.crt               controller0                        ServerAuth              Oct 10, 2026 12:30 UTC   1 year          OK
serving-kubelet.crt               k3s-server-ca@1760099372           CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client-k3s-controller.crt         system:k3s-controller              ClientAuth              Oct 10, 2026 12:30 UTC   1 year          OK
client-k3s-controller.crt         k3s-client-ca@1760099372           CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client-kube-apiserver.crt         system:apiserver                   ClientAuth              Oct 10, 2026 12:29 UTC   1 year          OK
client-kube-apiserver.crt         k3s-client-ca@1760099372           CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
serving-kube-apiserver.crt        kube-apiserver                     ServerAuth              Oct 10, 2026 12:29 UTC   1 year          OK
serving-kube-apiserver.crt        k3s-server-ca@1760099372           CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client-admin.crt                  system:admin                       ClientAuth              Oct 10, 2026 12:29 UTC   1 year          OK
client-admin.crt                  k3s-client-ca@1760099372           CertSign                Oct 08, 2035 12:29 UTC   10 years        OK

And after the rotation:

# /opt/bin/k3s certificate check -o table
INFO[0000] Server detected, checking agent and server certificates 

FILENAME                          SUBJECT                              USAGES                  EXPIRES                  RESIDUAL TIME   STATUS
--------                          -------                              ------                  -------                  -------------   ------
client-admin.crt                  system:admin                         ClientAuth              Oct 10, 2026 22:57 UTC   1 year          OK
client-admin.crt                  k3s-client-ca@1760135787             CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-admin.crt                  k3s-client-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-admin.crt                  k3s-client-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-admin.crt                  k3s-client-ca@1760099372             CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client-auth-proxy.crt             system:auth-proxy                    ClientAuth              Oct 10, 2026 22:57 UTC   1 year          OK
client-auth-proxy.crt             k3s-request-header-ca@1760135787     CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-auth-proxy.crt             k3s-request-header-root@1760135787   CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-auth-proxy.crt             k3s-request-header-root@1760135787   CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-auth-proxy.crt             k3s-request-header-ca@1760099372     CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client-scheduler.crt              system:kube-scheduler                ClientAuth              Oct 10, 2026 22:57 UTC   1 year          OK
client-scheduler.crt              k3s-client-ca@1760135787             CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-scheduler.crt              k3s-client-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-scheduler.crt              k3s-client-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-scheduler.crt              k3s-client-ca@1760099372             CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
kube-scheduler.crt                kube-scheduler                       ServerAuth              Oct 10, 2026 22:57 UTC   1 year          OK
kube-scheduler.crt                k3s-server-ca@1760135787             CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
kube-scheduler.crt                k3s-server-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
kube-scheduler.crt                k3s-server-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
kube-scheduler.crt                k3s-server-ca@1760099372             CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client-kube-proxy.crt             system:kube-proxy                    ClientAuth              Oct 10, 2026 22:57 UTC   1 year          OK
client-kube-proxy.crt             k3s-client-ca@1760135787             CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-kube-proxy.crt             k3s-client-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-kube-proxy.crt             k3s-client-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-kube-proxy.crt             k3s-client-ca@1760099372             CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client-kubelet.crt                system:node:controller0              ClientAuth              Oct 10, 2026 22:57 UTC   1 year          OK
client-kubelet.crt                k3s-client-ca@1760135787             CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-kubelet.crt                k3s-client-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-kubelet.crt                k3s-client-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-kubelet.crt                k3s-client-ca@1760099372             CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
serving-kubelet.crt               controller0                          ServerAuth              Oct 10, 2026 22:57 UTC   1 year          OK
serving-kubelet.crt               k3s-server-ca@1760135787             CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
serving-kubelet.crt               k3s-server-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
serving-kubelet.crt               k3s-server-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
serving-kubelet.crt               k3s-server-ca@1760099372             CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client-k3s-cloud-controller.crt   k3s-cloud-controller-manager         ClientAuth              Oct 10, 2026 22:57 UTC   1 year          OK
client-k3s-cloud-controller.crt   k3s-client-ca@1760135787             CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-k3s-cloud-controller.crt   k3s-client-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-k3s-cloud-controller.crt   k3s-client-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-k3s-cloud-controller.crt   k3s-client-ca@1760099372             CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client-controller.crt             system:kube-controller-manager       ClientAuth              Oct 10, 2026 22:57 UTC   1 year          OK
client-controller.crt             k3s-client-ca@1760135787             CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-controller.crt             k3s-client-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-controller.crt             k3s-client-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-controller.crt             k3s-client-ca@1760099372             CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
kube-controller-manager.crt       kube-controller-manager              ServerAuth              Oct 10, 2026 22:57 UTC   1 year          OK
kube-controller-manager.crt       k3s-server-ca@1760135787             CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
kube-controller-manager.crt       k3s-server-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
kube-controller-manager.crt       k3s-server-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
kube-controller-manager.crt       k3s-server-ca@1760099372             CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client.crt                        etcd-client                          ClientAuth              Oct 10, 2026 22:57 UTC   1 year          OK
client.crt                        k3s-etcd-server-ca@1760135787        CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client.crt                        k3s-etcd-server-root@1760135787      CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client.crt                        k3s-etcd-server-root@1760135787      CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client.crt                        etcd-server-ca@1760099372            CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
server-client.crt                 etcd-server                          ServerAuth,ClientAuth   Oct 10, 2026 22:57 UTC   1 year          OK
server-client.crt                 k3s-etcd-server-ca@1760135787        CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
server-client.crt                 k3s-etcd-server-root@1760135787      CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
server-client.crt                 k3s-etcd-server-root@1760135787      CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
server-client.crt                 etcd-server-ca@1760099372            CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
peer-server-client.crt            etcd-peer                            ServerAuth,ClientAuth   Oct 10, 2026 22:57 UTC   1 year          OK
peer-server-client.crt            k3s-etcd-peer-ca@1760135787          CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
peer-server-client.crt            k3s-etcd-peer-root@1760135787        CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
peer-server-client.crt            k3s-etcd-peer-root@1760135787        CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
peer-server-client.crt            etcd-peer-ca@1760099372              CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client-supervisor.crt             system:k3s-supervisor                ClientAuth              Oct 10, 2026 22:57 UTC   1 year          OK
client-supervisor.crt             k3s-client-ca@1760135787             CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-supervisor.crt             k3s-client-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-supervisor.crt             k3s-client-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-supervisor.crt             k3s-client-ca@1760099372             CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client-k3s-controller.crt         system:k3s-controller                ClientAuth              Oct 10, 2026 22:57 UTC   1 year          OK
client-k3s-controller.crt         k3s-client-ca@1760135787             CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-k3s-controller.crt         k3s-client-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-k3s-controller.crt         k3s-client-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-k3s-controller.crt         k3s-client-ca@1760099372             CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
client-kube-apiserver.crt         system:apiserver                     ClientAuth              Oct 10, 2026 22:57 UTC   1 year          OK
client-kube-apiserver.crt         k3s-client-ca@1760135787             CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-kube-apiserver.crt         k3s-client-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-kube-apiserver.crt         k3s-client-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
client-kube-apiserver.crt         k3s-client-ca@1760099372             CertSign                Oct 08, 2035 12:29 UTC   10 years        OK
serving-kube-apiserver.crt        kube-apiserver                       ServerAuth              Oct 10, 2026 22:57 UTC   1 year          OK
serving-kube-apiserver.crt        k3s-server-ca@1760135787             CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
serving-kube-apiserver.crt        k3s-server-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
serving-kube-apiserver.crt        k3s-server-root@1760135787           CertSign                Oct 05, 2045 22:36 UTC   20 years        OK
serving-kube-apiserver.crt        k3s-server-ca@1760099372             CertSign                Oct 08, 2035 12:29 UTC   10 years        OK

As mentioned in the K3S documentation (https://docs.k3s.io/cli/certificate#checking-expiration-dates), there are 2 lines for each certificate before the CA rotation. After this rotation, there are 5 lines for each certificate (due to the old CA, the cross-signed CA, the new CA, the intermediate CA and the leaf).

1. When can the old CA be deleted as well as the cross-signed CA?
2. How to verify that the new CA has been trusted everywhere it should have been?
3. Do you have a procedure to these "old" CAs? (a script or a command to avoid to remove old CA's manually in all files)

Thanks in advance

[brandond]

These are good questions, but the short answer is that the CA renewal script does not currently handle this. Once you have updated the token and restarted all your nodes, the old CA can be cleaned out of the bundles, and rotate-ca run again to load the new data - but there is not currently a script do to this cleanup for you.

A PR would be welcome, if anyone is interested in handling this.

[crtvmn]

What do you mean by "restarted all your nodes"? Restart k3s and k3s-agent service?

And you confirm that pods don’t have to be restarted?

Thank you very much for your answer!

[brandond]

Pods would need to be restarted as well, otherwise there's no guarantee that things will load the updated cluster CA bundles. If you know that everything you're running supports dynamic cert reloading then you could get away with just restarting the services, but I wouldn't bet on that being the case.