I found the memstrack tool and I want to report when kernel modules are loaded at boot because memstrack only collects memory data when you run it.
In dracut modules directory I found several files for memstrack:
ls -1 /usr/lib/dracut/modules.d/99memstrack/
memstrack-report.sh
memstrack.service
memstrack-start.sh
module-setup.sh
I don't understand how to use it. But I see in messages log that the memstrack service in systemd succeeded:
Aug 7 13:39:28 host01 systemd[1]: memstrack.service: Succeeded.
But:
systemctl status memstrack.service
Unit memstrack.service could not be found.
How it works? Will the report be generated only if the system crashes?
Also, I added rd.memdebug=4 to grub:
grep memdebug /etc/default/grub /boot/efi/EFI/rocky/grub.cfg
/etc/default/grub:GRUB_CMDLINE_LINUX="rd.memdebug=4 console=ttyS0,115200n8 no_timer_check crashkernel=auto"
/boot/efi/EFI/rocky/grub.cfg: set kernelopts="root=UUID=2b9b8a89-621d-4db3-86e4-202e84c2e50a ro rd.memdebug=4 console=ttyS0,115200n8 no_timer_check crashkernel=auto "
But I don't see this parameter in /proc/cmdline after boot:
cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt3)/boot/vmlinuz-4.18.0-372.19.1.el8_6.x86_64 root=UUID=2b9b8a89-621d-4db3-86e4-202e84c2e50a ro console=ttyS0,115200n8 no_timer_check crashkernel=auto
What I do wrong? Where I can see report about kernel modules memory usage generated by memstrack?
There is very little information about memstrack on the net.
If you know of a better tool for, please write. I tried to get information about the memory usage of kernel modules via slabinfo, but this is not quite what I need.
I want to get something similar to the screenshot:
But this screenshot taken when I ran memstrack and manually removed and loaded the kernel modules. I want to see this in real time without manually manipulating kernel modules.
UPD1: Yes, this report is generated when on system crash, but you need to rd.memdebug=[4,5] parameter to /etc/sysconfig/kdump in KDUMP_COMMANDLINE_APPEND. I checked this by simulating a crash with sysrq (echo c > /proc/sysrq-trigger):
sed -n '/module_summary/,/module_summary END/p' /var/crash/127.0.0.1-2022-08-07-15\:37\:53/kexec-dmesg.log
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: ======== Report format module_summary: ========
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module xfs using 16.0MB (4098 pages), peak allocation 34.4MB (8801 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module libcrc32c using 5.8MB (1474 pages), peak allocation 14.1MB (3610 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module squashfs using 4.9MB (1245 pages), peak allocation 4.9MB (1245 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module mgag200 using 3.4MB (861 pages), peak allocation 3.4MB (863 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module i2c_algo_bit using 0.9MB (238 pages), peak allocation 4.0MB (1022 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module smartpqi using 0.8MB (206 pages), peak allocation 1.8MB (469 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module overlay using 0.5MB (123 pages), peak allocation 0.6MB (163 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module sd_mod using 0.4MB (108 pages), peak allocation 1.5MB (373 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module syscopyarea using 0.4MB (94 pages), peak allocation 0.4MB (97 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module crc32c_intel using 0.4MB (90 pages), peak allocation 0.4MB (94 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module uas using 0.2MB (51 pages), peak allocation 0.2MB (54 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module scsi_transport_sas using 0.1MB (37 pages), peak allocation 0.1MB (37 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module t10_pi using 0.1MB (23 pages), peak allocation 0.1MB (23 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module drm_kms_helper using 0.1MB (20 pages), peak allocation 0.1MB (23 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module usb_storage using 0.1MB (19 pages), peak allocation 0.1MB (21 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module fb_sys_fops using 0.0MB (8 pages), peak allocation 0.0MB (8 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module drm using 0.0MB (8 pages), peak allocation 0.0MB (8 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module sysfillrect using 0.0MB (7 pages), peak allocation 0.0MB (7 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module sysimgblt using 0.0MB (7 pages), peak allocation 0.0MB (7 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module sg using 0.0MB (1 pages), peak allocation 0.0MB (1 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: Module hpwdt using 0.0MB (0 pages), peak allocation 0.0MB (0 pages)
Aug 07 15:38:00 host01 dracut-pre-pivot[490]: ======== Report format module_summary END ========
I still need a tool that will show how much RAM all kernel modules are currently using.
UPD2: I did the cron job for memstrack that creates a report after system boot:
cat /etc/cron.d/memstrack
@reboot root timeout 120 memstrack --report module_summary --notui --output /var/log/memstrack_report
cat /var/log/memstrack_report
======== Report format module_summary: ========
Module xfs using 363.7MB (93110 pages), peak allocation 403.2MB (103229 pages)
Module overlay using 22.6MB (5785 pages), peak allocation 22.6MB (5797 pages)
Module drm using 3.0MB (768 pages), peak allocation 3.0MB (768 pages)
Module sd_mod using 0.5MB (132 pages), peak allocation 0.5MB (132 pages)
Module openvswitch using 0.3MB (85 pages), peak allocation 0.3MB (85 pages)
Module kvm using 0.3MB (79 pages), peak allocation 0.3MB (79 pages)
Module nf_conntrack using 0.2MB (50 pages), peak allocation 0.2MB (50 pages)
Module geneve using 0.0MB (11 pages), peak allocation 0.0MB (11 pages)
Module nf_conncount using 0.0MB (8 pages), peak allocation 0.0MB (8 pages)
--->> Module ice using 0.0MB (8 pages), peak allocation 0.0MB (8 pages)
Module ip6_udp_tunnel using 0.0MB (8 pages), peak allocation 0.0MB (8 pages)
Module scsi_transport_iscsi using 0.0MB (4 pages), peak allocation 0.0MB (4 pages)
Module bonding using 0.0MB (2 pages), peak allocation 0.0MB (2 pages)
Module drm_kms_helper using 0.0MB (0 pages), peak allocation 0.0MB (0 pages)
Module udp_tunnel using 0.0MB (0 pages), peak allocation 0.0MB (0 pages)
But I an't satisfied with the result. Because, as you can see in /var/log/memstrack_report, the ice module used 8 pages and 0.0 mb, but in the screenshot above on line 2, when I loaded and removed the ice module manually, it used 5 GB of memory.