Ramdisk full errors

These days I had a problem with full ESXi ramdisk. Running vdf -h shows tmp as 100% used. Furthermore there are a lot of errors in vmkernel.log like:

VisorFSRam: 233: Cannot extend visorfs file /var/lib/vmware/hostd/journal/ because its ramdisk (root) is full.

There are some other problems these hosts suffer from time to time:

• vSphere HA issus like master of cluster cannot be found/elected.
• Connection problems with vCenter.

Hosts are ProLiant DL380 Gen8 servers running current ESXi 6.5 U2 image. After some troubleshooting I found out:

• /tmp/ql_ima.log consumed nearly all the space in tmp-partition.
• Some QLogic software logs into this file.
• ql_ima.log is locked by process hostd-worker.
• In ql_ima.log entries are about a libqima4xxx module. A component can be found in /usr/lib/vmware/ima_plugins named libqima4xxx.so.

Very strange fact about this findings is that there is no QLogic hardware running in this host! So there should be two ways to solve the problem: Update or uninstall appropriate driver. I decided to remove the unnecessary software. I searched for driver by running:

esxcli software vib list | grep -i ima

and found: ima-qla4xxx; version: 500.2.01.31-1vmw.0.3.040121. To remove it, I run:

esxcli software vib remove -n ima-qla4xxx

After a reboot:

• no /tmp/ql_ima.log any more.
• no /usr/lib/vmware/ima_plugins/libqima4xxx.so any more.
• temp-partition uses less than 200kb.

Problem removed.

Notes

• To remove a VIB and use Update Manager to patch a host without rebooting does not work. After remove a driver, Update Manager scan shows all patches of the current version (6.5) as missing.
• When your host suffers this issue, try to restart management agents. Without it could happens your host does not even reboot gracefully because of full ramdisk.