Certificate Rotation in K3s— Confirmation and Clarifications Needed

[AshiqN]

Hi Team,

We are currently building a CLI tool to deploy and manage our application on on-premise customer infrastructure, with K3s as our chosen Kubernetes distribution. Since K3s is fully managed by our CLI, we’re extending its capabilities to support certificate rotation through the commands:

• mycli certificate rotate
• mycli certificate rotate-ca

Below, I’ve outlined the implementation steps we’ve developed for both leaf certificate rotation and CA certificate rotation (self-signed and custom). We would appreciate your confirmation on whether these are correct, and we also have a few specific questions.

---

k3s version v1.32.6+k3s1 (eb603ac)
go version go1.23.10

---

Leaf Certificate Rotation

(Command: mycli certificate rotate)

Steps executed on each node (currently only server nodes; agent nodes will be added soon):

1. Run: k3s certificate rotate
2. Restart the K3s service: sudo systemctl restart k3s

These steps are repeated on each server node.

Question 1:
Are these steps sufficient for rotating leaf certificates on server nodes? Additionally, can I follow the same process for rotating certificates on agent nodes, or do agent nodes require a different approach?
The official documentation mentions:

1. systemctl stop k3s
2. k3s certificate rotate
3. systemctl start k3s
   Can we safely replace these with a simpler k3s certificate rotate followed by systemctl restart k3s?

---

CA Certificate Rotation (Self-Signed CA)

(Command: mycli certificate rotate-ca)

We are currently using the default self-signed CA generated by K3s. To rotate it, we follow these steps:

1. Generate new CA certificates using the official rotate ca cert script from the K3s documentation and store them in a temporary directory.
2. Run: k3s certificate rotate-ca --path <temp_dir>
3. Retrieve the current node token: cat <data-dir>/server/node-token, and update all nodes' config.yaml files with this token.
4. Rotate leaf certificates again via: k3s certificate rotate
   (Note: This step is not mentioned in the docs, but omitting it caused the cluster to fail during a second CA rotation with trust issues.)
5. Restart K3s: sudo systemctl restart k3s
6. Repeat steps 4 and 5 on all nodes.
7. Restart all application pods.

Question 2:
Are these the correct and complete steps for rotating self-signed CA certificates?

Question 3:
Regarding the node token:

• Can the current node-token be reused after the rotate-ca operation?
• Or should the node be restarted first to generate a new token?
• What is the difference in /var/lib/rancher/k3s/server/node-token and /var/lib/rancher/k3s/server/token

---

CA Certificate Rotation (Custom CA – Same Root CA)

When rotating a custom CA while keeping the same root CA, our current approach is:

1. Generate updated certificates using the k3s generate custom ca script and save them to a temporary directory.
2. Run: k3s certificate rotate-ca --path <temp_dir>
3. Restart the K3s service on all nodes.

Question 4:
If the root CA itself has changed, what is the correct procedure for rotating the CA certificates?
We attempted the documented approach, but it did not work as expected. Any guidance on the proper steps would be appreciated.

Question 5:
Is it possible to switch between a self-signed CA and a custom CA, and vice versa?
If so, what is the safest and cleanest way to perform such a transition without breaking trust within the cluster?

---

Thank you for your time and support. We're looking forward to your feedback so we can ensure our CLI tool handles certificate rotation reliably and in alignment with best practices.

[brandond]

I've personally validated the steps from the docs, so any deviations from that (such as running k3s certificate rotate to manually rotate leaf certificates after running k3s certificate rotate-ca) shouldn't be necessary. The leaf certs will automatically get regenerated if the CA certs change, if you're seeing otherwise then I suspect something else is being missed in the previous steps. I'm also not sure what you mean by "fail during a second CA rotation". Are you trying to rotate multiple times, without restarting all the nodes in between?

Note that the join token only needs to be updated if you change the server CA trust anchor. As long as the root CA for the server CA bundle remains the same, the CA hash in the token will remain the same. This is covered in the CA topology diagrams in the CA rotation docs, and in the token format docs.

If you're looking for more hands-on assistance with validating a product built around K3s, you might consider becoming a partner: https://www.suse.com/partners/. As a community project, we're not really staffed to validate products built on top of K3s. The partner team would be able to allocate more resources to work directly with you to validate what you're trying to do.

[AshiqN]

How many nodes are in your cluster when you do this?
One node. I also tested with multi-node cluster.

What version are you testing with?
k3s version v1.32.6+k3s1 (eb603ac)
go version go1.23.10

Are you waiting for K3s to finish starting completely before rotating the CA certs again?
Yes, after each rotation I verify using the certificate check command and review the K3s logs to confirm it's fully started.

Do you see different results if you remove the content from /var/lib/rancher/k3s/server/rotate-ca after step 3, before re-running the rotation?
Yes, sometimes it works, and other times it doesn't.

As a temporary workaround, I am rotating the leaf certificates as well. The another reason for this is that if I restart without rotating the leaf certificates, the node token I receive remains the old one, which causes issues when configuring nodes using that outdated token.
So, is it acceptable to rotate the leaf certificates after rotating the CA certificates?

I’d be glad to contribute a fix for this, if I’m able to.

[brandond]

There shouldn't be anything non-deterministic about this process. Can you identify any difference in your process between when it works and when it doesn't?

[AshiqN]

Hi @brandond ,
I noticed why the leaf certificates are not being regenerated. It seems K3s only regenerates leaf certificates when their expiry is within 90 days. If someone triggers the rotate-ca command before that period and restarts the server, it can cause issues.
I’ve raised a PR that addresses this behavior.
#12978

[brandond]

It should regenerate all the static leaf certificates if their CA changes regardless of their expiry time. The dynamiclistener cert does have slightly different behavior.

[AshiqN]

Then I’m not entirely sure why this issue is occurring, as it is still reproducible on my end. However, if we regenerate the leaf certificates before restarting, the failure does not occur.
The failure happens here during restart: token.go#L434

Using the regenerate-file approach resolves the issue—I was able to rotate certificates multiple times without any problems.