Ramdisk full errors

These days I had a problem with full ESXi ramdisk. Running vdf -h shows tmp as 100% used. Furthermore there are a lot of errors in vmkernel.log like:

VisorFSRam: 233: Cannot extend visorfs file /var/lib/vmware/hostd/journal/ because its ramdisk (root) is full.

There are some other problems these hosts suffer from time to time:

  • vSphere HA issus like master of cluster cannot be found/elected.
  • Connection problems with vCenter.

Hosts are ProLiant DL380 Gen8 servers running current ESXi 6.5 U2 image. After some troubleshooting I found out:

  • /tmp/ql_ima.log consumed nearly all the space in tmp-partition.
  • Some QLogic software logs into this file.
  • ql_ima.log is locked by process hostd-worker.
  • In ql_ima.log entries are about a libqima4xxx module. A component can be found in /usr/lib/vmware/ima_plugins named libqima4xxx.so.

Very strange fact about this findings is that there is no QLogic hardware running in this host! So there should be two ways to solve the problem: Update or uninstall appropriate driver. I decided to remove the unnecessary software. I searched for driver by running:

esxcli software vib list | grep -i ima

and found: ima-qla4xxx; version: 500.2.01.31-1vmw.0.3.040121. To remove it, I run:

esxcli software vib remove -n ima-qla4xxx

After a reboot:

  • no /tmp/ql_ima.log any more.
  • no /usr/lib/vmware/ima_plugins/libqima4xxx.so any more.
  • temp-partition uses less than 200kb.

Problem removed.

Notes

  • To remove a VIB and use Update Manager to patch a host without rebooting does not work. After remove a driver, Update Manager scan shows all patches of the current version (6.5) as missing.
  • When your host suffers this issue, try to restart management agents. Without it could happens your host does not even reboot gracefully because of full ramdisk.

Leave a Reply

Your email address will not be published. Required fields are marked *