In the case where you need to diagnose kernel crash issues on a BCM managed cluster based on RHEL you can use the kexec tools that are part of the underlying operating system. These instructions have been tested against BCM 10 and RHEL8 but they should remain similar across a wide variety of RHEL operating systems and BCM versions.
These instructions are a supplement to the operating systems procedures, some of those are linked below.
These instructions will demonstrate dumping over NFS and SSH, please refer to operating system documentation for more options.
Install required packages
Install the kexec-tools in the software image. Please complete these steps on all images where you will need the kdump service running.
dnf install kexec-tools --installroot=/cm/images/<image-name>
Modify the software image
Configure the software image to allow crashkernel memory. Please refer to operating system documentation above on how to tune the crashkernel command line variable for your particular system. Failure to set this to an appropriate size may result in silent failures of the kdump system.
NOTE: The crashkernel=auto option in the boot command line is no longer supported on RHEL 9 and later releases.
# cmsh
% softwareimage use <image-name>
% append kernelparameters " crashkernel=512M" # please make sure that the append starts with a blank space
% commit
Configure kdump in the software image
We must first alter the /etc/kdump.conf inside the image to use our preferred method of dumping the image. These changes will be different depending on what protocol we use. You may also use other targets but we have found that locally targeted kdump may be overwritten by the node-installer and therefore writing it out over the network may provide a more reliable target.
Kdump over NFS
Please alter the /etc/kdump.conf inside the image with the following parameters and comment out any other settings.
nfs master.cm.cluster:/cm/shared/kdump
path /
core_collector makedumpfile -l --message-level 7 -d 31
With these settings, the kernel dumps will be written to /cm/shared/kdump/<IP>-<time> on master.cm.cluster. Please make sure the path /cm/shared/kdump/ is present on the NFS server.
Kdump over SSH
Please alter the /etc/kdump.conf inside the image with the following parameters and comment out any other settings. The ssh key that I am using in my test the ssh key from cmsupport coped to the path indicated within the software image.
ssh cmsupport@master.cm.cluster
sshkey /root/.ssh/kdump_ssh_key
path /tmp
core_collector makedumpfile -F -l --message-level 7 -d 31
With this configuration, files will be scp’ed from the kdump host to master.cm.cluster in the path /tmp/<IP>-<time> with the ownership of cmsupport. Please note the addition of the -F in the core_collector option.
Kdump to logical volume or partition
If network solutions are not appropriate you can use a fixed, local logical volume or partition would be a better option. We will use the example below of a logical volume but a partition with a label would also work. We will keep this filesystem unmounted so that it should not be affected by a resync but if the device order changes you will need to set datanode to yes to prevent reformating the node.
Create the disk layout. One example of this is below, this should be used as an example and should be altered to fit your needs.
<?xml version="1.0" encoding="UTF-8"?>
<diskSetup>
<device>
<blockdev>/dev/vdb</blockdev>
<blockdev>/dev/sda</blockdev>
<blockdev>/dev/nvme0n1</blockdev>
<blockdev mode="cloud">/dev/oracleoci/oraclevdb</blockdev>
<blockdev mode="cloud">/dev/sdb</blockdev>
<blockdev mode="cloud">/dev/hdb</blockdev>
<blockdev mode="cloud">/dev/vdb</blockdev>
<blockdev mode="cloud">/dev/xvdf</blockdev>
<partition id="a1">
<size>512M</size>
<type>linux</type>
<filesystem>ext2</filesystem>
<mountPoint>/boot</mountPoint>
<mountOptions>defaults,noatime,nodiratime</mountOptions>
</partition>
<partition id="a2">
<size>1G</size>
<type>linux swap</type>
</partition>
<partition id="a3">
<size>max</size>
<type>linux lvm</type>
</partition>
</device>
<volumeGroup>
<name>vg01</name>
<extentSize>8M</extentSize>
<physicalVolumes>
<member>a3</member>
</physicalVolumes>
<logicalVolumes>
<volume>
<name>kdump</name>
<size>2G</size>
</volume>
<volume>
<name>lv00</name>
<size>max</size>
<filesystem>xfs</filesystem>
<mountPoint>/</mountPoint>
<mountOptions>defaults,noatime,nodiratime</mountOptions>
</volume>
</logicalVolumes>
</volumeGroup>
</diskSetup>
The kdump partition should be at least 2GB for proper operation. Once this has been applied to the host you can then see and format the logical volume.
[root@node001 ~]# mkfs.ext4 -L kdump /dev/vg01/kdump
mke2fs 1.45.6 (20-Mar-2020)
/dev/vg01/kdump contains a ext4 file system labelled 'kdump'
created on Wed Oct 30 16:07:57 2024
Proceed anyway? (y,N) y
Discarding device blocks: done
Creating filesystem with 524288 4k blocks and 131072 inodes
Filesystem UUID: ac894da9-bb43-4107-93ff-21cd4d98e359
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912
Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
By applying the label the following kdump.conf should work with logical volumes or fixed partitions.
ext4 LABEL=kdump
#ext4 /dev/vg01/kdump
path /
core_collector makedumpfile -l --message-level 7 -d 31
After a crash you can mount the filesystem to review the vmcore.
[root@node001 ~]# mount /dev/vg01/kdump /mnt
[root@node001 ~]# ls -al /mnt
total 28
drwxr-xr-x 4 root root 4096 Oct 30 16:37 .
dr-xr-xr-x. 21 root root 4096 Oct 11 17:33 ..
drwxr-xr-x 2 root root 4096 Oct 30 16:38 127.0.0.1-2024-10-30-16:37:58
drwx------ 2 root root 16384 Oct 30 16:25 lost+found
[root@node001 ~]# ls -al /mnt/127.0.0.1-2024-10-30-16\:37\:58/
total 54168
drwxr-xr-x 2 root root 4096 Oct 30 16:38 .
drwxr-xr-x 4 root root 4096 Oct 30 16:37 ..
-rw------- 1 root root 46329 Oct 30 16:38 kexec-dmesg.log
-rw------- 1 root root 55367417 Oct 30 16:38 vmcore
-rw------- 1 root root 39753 Oct 30 16:37 vmcore-dmesg.txt
[root@node001 ~]#
Enable kdump in image
For each image where kdump will run please enable the service.
# cm-chroot-sw-img <path to image>
# systemctl enable kdump
Confirm kdump is started and configured
Boot one of the nodes and confirm that kdump service is started without errors
[root@node001 ~]# cat /proc/cmdline
BOOT_IMAGE=images/default-image/vmlinuz initrd=images/default-image/initrd crashkernel=auto console=tty0 ip=10.141.0.1:10.141.255.254:10.141.255.254:255.255.0.0 BOOTIF=01-fa-16-3e-38-ca-09
[root@node001 ~]# systemctl status kdump
● kdump.service - Crash recovery kernel arming
Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; vendor preset: enabled)
Active: active (exited) since Wed 2024-10-16 19:29:43 CEST; 10s ago
Process: 9050 ExecStart=/usr/bin/kdumpctl start (code=exited, status=0/SUCCESS)
Main PID: 9050 (code=exited, status=0/SUCCESS)
Oct 16 19:29:30 node001 dracut[9229]: *** Install squash loader ***
Oct 16 19:29:31 node001 dracut[9229]: *** Stripping files ***
Oct 16 19:29:31 node001 dracut[9229]: *** Stripping files done ***
Oct 16 19:29:31 node001 dracut[9229]: *** Squashing the files inside the initramfs ***
Oct 16 19:29:42 node001 dracut[9229]: *** Squashing the files inside the initramfs done ***
Oct 16 19:29:42 node001 dracut[9229]: *** Creating image file '/boot/initramfs-4.18.0-513.5.1.el8_9.x86_64kdump.img' ***
Oct 16 19:29:42 node001 dracut[9229]: *** Creating initramfs image file '/boot/initramfs-4.18.0-513.5.1.el8_9.x86_64kdump.img' done ***
Oct 16 19:29:43 node001 kdumpctl[9052]: kdump: kexec: loaded kdump kernel
Oct 16 19:29:43 node001 kdumpctl[9052]: kdump: Starting kdump: [OK]
Oct 16 19:29:43 node001 systemd[1]: Started Crash recovery kernel arming.
The important parts above is the inclusion of crashkernel in the /proc/cmdline and the Crash recovery kernel arming. in the kdump status.
Generated a kernel panic to test.
Once the above steps are created you can use the following to trigger a core dump
# ssh <node>
# # WARNING, THE NEXT COMMAND WILL CRASH NODE
#echo c > /proc/sysrq-trigger
You can then check the target locations defined above to see if a crash has been successfully generated.