Optimizing Log Storage on Devices with SD Cards
On devices that utilise an SD card for their file system such as the Raspberry Pi, card wear emerges as a significant concern. Linux tends to generate an extensive log of information in /var/log, which, over time, leads to a high volume of disk writes and ultimately shortens the device's lifespan. While this is a well-known issue, the proper and permanent solution is less commonly understood.
Instead of suggesting the use of 'tmpfs' or 'ramfs' to mount at '/var/log', which presents two primary drawbacks:
- It consumes system RAM, which can be severely limited on small embedded devices.
- It can fill up and impede the proper functioning of services.
A more effective solution can be found within the kernel itself. During the boot process, when no file system is available for logging, the kernel already possesses a ring buffer, a memory-based message buffer. Additionally, if a storage failure occurs, the kernel still requires a location to output informational messages for diagnostic purposes. As the name implies, this buffer operates in a circular manner, overwriting itself from the start when it becomes full.
We can leverage this existing mechanism by configuring 'rsyslog' to write directly to the kernel's ring buffer instead of a file, and the same can be done for the 'systemd' journal.
To redirect all logging to the kernel's ring buffer, replace '/etc/rsyslog.conf' with the following:
# provides support for local system logging
# keep the output short and simple
# send everything to the kernel's kmsg
# Configure imuxsock module for local system logging (e.g., via logger command)
To configure the 'systemd' journal to also utilise the kernel's ring buffer, make the following changes in '/etc/systemd/journald.conf':
After implementing these modifications, either reboot the device or execute the following commands:
systemctl restart rsyslog
systemctl restart systemd-journald
To view the output, use the command
dmesg to display the contents of the kernel's ring buffer, or employ
dmesg -w to continuously monitor it.
Note that it still may be prudent to mount a `tmpfs` at `/var/log` for services that write here directly.