I’m running Alma9 on a semi-embedded system with limited storage. I’d like to find out what units’ logs use most space.
Unfortunately the journalctl --disk-usage doesn’t respect the --unit <unit> flag.But journalctl --unit <unit> | wc -c ought to at least be roughly proportional to the disk space used.
So, I tried
for u in $(journalctl --field=UNIT)
do
printf "%10d %s\n" $(journalctl --unit $u | wc -c) $u
done | sort -n
But - this misses lots of services! It turns out that journalctl --field=UNIT does NOT show all units!
There are a bunch of services, such as itself — systemd-journald.service that it doesn’t show!
At the same time, it appears that the bash completion for journalctl --unit [tab] is able to detect that there is such a unit as systemd-journald.service.
I’ve been trying to parse the /usr/share/bash-completion/completions/journalctl completion script - but it’s not the most readable. (By no means the worst completion script I’ve ever seen, though.)
So - my question is: How can I find out what units have logged logs in the journald journal?
and ideally also How can I find out how much disk space is taken up by these units?
The systemd-journald does not supposedly store anything on the disk by default.
The persistent logging would go to /var/log/journal/ by default, but that directory is not created by default install.
If the persistent store is enabled, the logs are still not “by unit”.
total 672M
-rw-r-----+ 1 root systemd-journal 48M Mar 31 13:18 system.journal
-rw-r-----+ 1 root systemd-journal 56M Mar 16 02:58 system@51d0796cd6df41faa6ee8989e2f56291-0000000000000001-00062ff7f0b7e0c3.journal
-rw-r-----+ 1 root systemd-journal 56M Mar 18 10:26 system@51d0796cd6df41faa6ee8989e2f56291-000000000001fdf9-0006306b2b7476ec.journal
-rw-r-----+ 1 root systemd-journal 56M Mar 20 21:52 system@51d0796cd6df41faa6ee8989e2f56291-000000000003dad0-00063099a7c5444a.journal
-rw-r-----+ 1 root systemd-journal 56M Mar 23 04:23 system@51d0796cd6df41faa6ee8989e2f56291-000000000005ba4a-000630cb7ae43df9.journal
-rw-r-----+ 1 root systemd-journal 56M Mar 25 07:37 system@51d0796cd6df41faa6ee8989e2f56291-000000000007942a-000630f92ca69a1d.journal
-rw-r-----+ 1 root systemd-journal 56M Mar 27 12:03 system@51d0796cd6df41faa6ee8989e2f56291-0000000000097539-000631241e2e1ec5.journal
-rw-r-----+ 1 root systemd-journal 56M Mar 29 18:23 system@51d0796cd6df41faa6ee8989e2f56291-00000000000b416b-0006315010475b78.journal
-rw-r-----+ 1 root systemd-journal 40M Mar 31 13:17 user-1911.journal
-rw-r-----+ 1 root systemd-journal 48M Mar 21 15:22 user-1911@6a7b29c049b345a5a5572637b73539b9-000000000000755a-0006300e4acb98b7.journal
-rw-r-----+ 1 root systemd-journal 48M Mar 27 02:32 user-1911@6a7b29c049b345a5a5572637b73539b9-000000000006521d-000630da264907df.journal
-rw-r-----+ 1 root systemd-journal 16M Mar 31 13:16 user-85992.journal
-rw-r-----+ 1 root systemd-journal 72M Mar 20 05:15 user-85992@0a4999adea0b4000bafb43d7d199760c-0000000000000833-00062ff8146bf4a1.journal
-rw-r-----+ 1 root systemd-journal 72M Mar 30 02:15 user-85992@0a4999adea0b4000bafb43d7d199760c-00000000000536dd-000630bd8b2ade83.journal
Have you enabled the persistent store?
What does write to disk by default is ‘rsyslog.service’. It merely copies some messages from journald to its legacy format files. Furthermore, package rsyslog-logrotate is not installed by default, so rsyslog’s files can grow forever.
I’d start with doing something to rsyslog (if you still have it) – either disable completely, or at least rotate and not keep forever.
I’d guess your question is: Which units are unnecessarily verbose?
Knowing that one could choose to tell some be quiet.
Alas, I have not looked at that.
Note. The journalctl --field=UNIT seems to list session scopes too. (E.g. separate scope for each cron job, etc.)
Sorry - I should have been more explicit. Yes - I do create the /var/log/journal.
And journalctl --disk-usage tells me that I have ~100MB of logs on disk.
I explicitly don’t want to fill the disks with logs, so I only use systemd-journald, and set it to limit the on-disk logs to ~100MByte, which corresponds to about three or four days of normal use. Unfortunately the journald logs are in a rather space-consuming proprietary format¹, which is optimized for read access, rather than for space.
(I can compress the logfiles to ~4% of the space, using zstd -19.)
Yes - systemctl --field=UNIT does indeed list more than just services. But what I’m worried about is that it doesn’t seem to list all the services that have generated logs.
For example:
journalctl --field=UNIT | grep systemd-journald
shows nothing, even though
journalctl --unit=systemd-journald.service
shows a couple of lines about the journald.
¹ And that is with Compress=yes in the /etc/systemd/journald. Apparently it compresses only certain fields, rather than the entire file. With compression enabled, 56% of all bytes in the log files are ‘NUL’!
Compress=
Can take a boolean value. If enabled (the default), data objects that shall be stored in the journal and are larger than the default threshold of 512 bytes are compressed before they are written to the file system. It can also be set to a number of bytes to specify the compression threshold directly.
I.e. lines shorter than 512 bytes remain uncompressed. Implies that compression is “per-line” (or some fields of a line).
That seems to be a trend – “verbose” machine-actionable (meta)data.
Interestingly, my server does list the systemd-journald.service that yours omits:
# journalctl -N | grep UNIT
UNIT_RESULT
UNIT
DBUS_BROKER_LAUNCH_SERVICE_UNIT
_SYSTEMD_UNIT
_SYSTEMD_USER_UNIT
USER_UNIT
# journalctl -F UNIT | grep jour
systemd-journal-catalog-update.service
systemd-journal-flush.service
systemd-journald.service
# journalctl -F _SYSTEMD_UNIT | grep jour
systemd-journald.service
# journalctl -F _SYSTEMD_USER_UNIT | grep jour
# journalctl -F USER_UNIT | grep jour
#