Is it normal to have a SQLite datastore state.db of 5Gb in size?

[gerardnico]

Hallo,

I don't know if this is normal (all documentation says that it should be around 500mb) but the sqlite datastore is 5gb in size.

On the kine table, there is a lot of data for cert-manager. The analytics rows (see below) shows

COUNT(*)  name
--------  ------------------------------------------------------------
278873    /registry/leases/cert-manager/trust-manager-leader-election
49956     /registry/leases/kube-system/cert-manager-controller
49921     /registry/leases/kube-system/cert-manager-cainjector-leader-election

but I don't know if I can minimize it and if I should

May be the answer is that there is nothing to do or I should get my certificates in another way.

Thanks

More info

Version

• k3s version: v1.33.5+k3s1 (fab4a5c)

What did I do

I already restart/vacuum it.
I went from 12Gb to 5GB.

Size

du -sh /var/lib/rancher/k3s/server/db/*

8.0K    /var/lib/rancher/k3s/server/db/etcd
5.1G    /var/lib/rancher/k3s/server/db/state.db
32K     /var/lib/rancher/k3s/server/db/state.db-shm
7.6M    /var/lib/rancher/k3s/server/db/state.db-wal

The analytics query

sqlite3 /var/lib/rancher/k3s/server/db/state.db << EOF
.header on
.mode column
.dbinfo


SELECT COUNT(*), name FROM kine GROUP BY name ORDER BY COUNT(*) DESC LIMIT 50;

EOF

output

COUNT(*)  name
--------  ------------------------------------------------------------
278873    /registry/leases/cert-manager/trust-manager-leader-election
77684     /registry/masterleases/x.245.43.249
71997     /registry/leases/kube-system/apiserver-fohhc347juam3arofccezbofla
71962     /registry/leases/kube-node-lease/kube-server-01.xxxx.com
49956     /registry/leases/kube-system/cert-manager-controller
49921     /registry/leases/kube-system/cert-manager-cainjector-leader-election
5229      /registry/cronjobs/mail/mail-checker-cron
4338      /registry/argoproj.io/applications/argocd/com-datacad
...

[brandond]

Check the logs for compact messages. You should see compact running, or erroring, every 5 minutes or so.

[gerardnico]

Yeah thanks. Forgot to say that it’s working. And I already run your delete query to see if I was gaining anything but it seems that it’s genuine data. It’s for now, a single box cluster with a 40 pods. I don’t know what kind of data is generated and if I can have any impact on them.

K3s is behaving correctly but with 5g data store it takes sometimes to backup it.

[brandond]

Those counts are all really high. If compaction is working there should not be that many entries present for any key.

What do the compact logs show?

[gerardnico]

Thanks.

I did already a manual vacuum and executing the delete statement, thinking that it was done.

I checked again with sqlite_analyzer and I got again a big difference between the file size and the payload

Size of the file in bytes......................... 5368762368
Bytes of user payload stored...................... 67211751     1.3%

I executed again a manual vacuum and boum, pretty small now.

8.0K    /var/lib/rancher/k3s/server/db/etcd
71M     /var/lib/rancher/k3s/server/db/state.db
32K     /var/lib/rancher/k3s/server/db/state.db-shm
4.3M    /var/lib/rancher/k3s/server/db/state.db-wal

Thanks.

[gerardnico]

The compact log.

journalctl -e -u k3s --since "15 minutes ago" | grep -i compact

Oct 02 09:29:48 kube-server-01.example.com k3s[2375638]: time="2025-10-02T09:29:48Z" level=info msg="COMPACT compactRev=45932240 targetCompactRev=45932638 currentRev=45933638"
Oct 02 09:29:48 kube-server-01.example.com k3s[2375638]: time="2025-10-02T09:29:48Z" level=info msg="COMPACT deleted 398 rows from 398 revisions in 146.420287ms - compacted to 45932638/45933638"
Oct 02 09:29:48 kube-server-01.example.com k3s[2375638]: time="2025-10-02T09:29:48Z" level=info msg="COMPACT compacted from 45932240 to 45932638 in 1 transactions over 147ms"
Oct 02 09:34:48 kube-server-01.example.com k3s[2375638]: time="2025-10-02T09:34:48Z" level=info msg="COMPACT compactRev=45932638 targetCompactRev=45933010 currentRev=45934010"
Oct 02 09:34:48 kube-server-01.example.com k3s[2375638]: time="2025-10-02T09:34:48Z" level=info msg="COMPACT deleted 372 rows from 372 revisions in 27.101136ms - compacted to 45933010/45934010"
Oct 02 09:34:48 kube-server-01.example.com k3s[2375638]: time="2025-10-02T09:34:48Z" level=info msg="COMPACT compacted from 45932638 to 45933010 in 1 transactions over 28ms"
Oct 02 09:39:48 kube-server-01.example.com k3s[2375638]: time="2025-10-02T09:39:48Z" level=info msg="COMPACT compactRev=45933010 targetCompactRev=45933417 currentRev=45934417"
Oct 02 09:39:49 kube-server-01.example.com k3s[2375638]: time="2025-10-02T09:39:49Z" level=info msg="COMPACT deleted 407 rows from 407 revisions in 154.625534ms - compacted to 45933417/45934417"
Oct 02 09:39:49 kube-server-01.example.com k3s[2375638]: time="2025-10-02T09:39:49Z" level=info msg="COMPACT compacted from 45933010 to 45933417 in 1 transactions over 155ms"

[brandond]

Are your row counts lower now that you've run that delete query to manually remove old versions? What did the compact logs show before doing that?

[gerardnico]

Sorry for the late answer but I took a bad charger for my laptop this weekend.

I took a look one week ago and I can't find any compaction.

I don't remember exactly every step that I took (I was also out of disk space) but it seems that I have:

• run sql_analyzer
• saw that I needed to vacuum it
• vacuumed it
• run your delete statement

delete from kine where id in (select id from (select id, name from kine where id not in (select max(id) as id from kine group by name)));

• started the whole thing again
• checked compaction

It seems that the delete statement was:

• of not successful
• of is not the full compaction
• of ...

Thanks
Have a nice week

One week ago journal

journalctl -u k3s --since "7 days ago" --until "6 days 23 hours 45 minutes ago"

Sep 29 06:35:28 kube-server-01.example.com k3s[4146498]: I0929 06:35:28.557372 4146498 metrics.go:299] "Failed to get storage metrics" storage_cluster_id="etcd-0" err="context deadline exceeded"
Sep 29 06:35:32 kube-server-01.example.com k3s[4146498]: I0929 06:35:32.863954 4146498 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbc130eb-438b-48b7-817b-0c0e198ff472" path="/var/lib/kubelet/pods/fbc130eb-438b-48b7-817b-0c0e19>
Sep 29 06:35:33 kube-server-01.example.com k3s[4146498]: I0929 06:35:33.066719 4146498 job_controller.go:598] "enqueueing job" logger="job-controller" key="mail/mail-checker-cron-29318795" delay="0s"
Sep 29 06:36:07 kube-server-01.example.com k3s[4146498]: I0929 06:36:07.514503 4146498 metrics.go:299] "Failed to get storage metrics" storage_cluster_id="etcd-0" err="context deadline exceeded"
Sep 29 06:36:13 kube-server-01.example.com k3s[4146498]: I0929 06:36:13.593296 4146498 metrics.go:299] "Failed to get storage metrics" storage_cluster_id="etcd-0" err="context deadline exceeded"
Sep 29 06:36:13 kube-server-01.example.com k3s[4146498]: I0929 06:36:13.923644 4146498 scope.go:117] "RemoveContainer" containerID="23ce0fa46e0c5e2898bdc99b90d69d103255ae15fd01dab8084c79143741ef23"
Sep 29 06:36:58 kube-server-01.example.com k3s[4146498]: I0929 06:36:58.595094 4146498 metrics.go:299] "Failed to get storage metrics" storage_cluster_id="etcd-0" err="context deadline exceeded"
Sep 29 06:37:07 kube-server-01.example.com k3s[4146498]: I0929 06:37:07.499043 4146498 metrics.go:299] "Failed to get storage metrics" storage_cluster_id="etcd-0" err="context deadline exceeded"
Sep 29 06:37:43 kube-server-01.example.com k3s[4146498]: I0929 06:37:43.658144 4146498 metrics.go:299] "Failed to get storage metrics" storage_cluster_id="etcd-0" err="context deadline exceeded"
Sep 29 06:38:07 kube-server-01.example.com k3s[4146498]: I0929 06:38:07.544268 4146498 metrics.go:299] "Failed to get storage metrics" storage_cluster_id="etcd-0" err="context deadline exceeded"
Sep 29 06:38:28 kube-server-01.example.com k3s[4146498]: I0929 06:38:28.577853 4146498 metrics.go:299] "Failed to get storage metrics" storage_cluster_id="etcd-0" err="context deadline exceeded"
Sep 29 06:39:07 kube-server-01.example.com k3s[4146498]: I0929 06:39:07.604199 4146498 metrics.go:299] "Failed to get storage metrics" storage_cluster_id="etcd-0" err="context deadline exceeded"
Sep 29 06:39:13 kube-server-01.example.com k3s[4146498]: I0929 06:39:13.549695 4146498 metrics.go:299] "Failed to get storage metrics" storage_cluster_id="etcd-0" err="context deadline exceeded"
Sep 29 06:39:55 kube-server-01.example.com k3s[4146498]: I0929 06:39:55.266777 4146498 range_allocator.go:247] "Successfully synced" key="kube-server-01.example.com"
Sep 29 06:39:58 kube-server-01.example.com k3s[4146498]: I0929 06:39:58.757997 4146498 metrics.go:299] "Failed to get storage metrics" storage_cluster_id="etcd-0" err="context deadline exceeded"