Found invalid object errors on storage volumes

I want to share the solution of flooding errors I had after an vSphere upgrade. Errors are probably not connected to upgrade.

The problem

I got flooded by errors for VMFS datastores in log files.

Errors in /var/log/vmkernel.log:

2020-01-29T14:06:00.311Z cpu8:2099233)WARNING: Fil3: 8175: Found invalid object on 59c4eb22-52356382-42b5-5cb90195e9c0 expected
2020-01-29T14:06:00.311Z cpu8:2099233)Vol3: 3374: Failed to get object 28 type 3 uuid 59c4eb22-52356382-42b5-5cb90195e9c0 FD 8029c4 gen 3 :Not found
2020-01-29T14:06:00.312Z cpu8:2099233)WARNING: Fil3: 8175: Found invalid object on 59c4eb22-52356382-42b5-5cb90195e9c0 expected
2020-01-29T14:06:00.312Z cpu8:2099233)Vol3: 3374: Failed to get object 28 type 3 uuid 59c4eb22-52356382-42b5-5cb90195e9c0 FD 8029c4 gen 3 :Not found
2020-01-29T14:06:00.312Z cpu13:2099233)WARNING: Fil3: 8175: Found invalid object on 59c4eb22-52356382-42b5-5cb90195e9c0 expected
2020-01-29T14:06:00.312Z cpu13:2099233)Vol3: 3374: Failed to get object 28 type 3 uuid 59c4eb22-52356382-42b5-5cb90195e9c0 FD 8029c4 gen 3 :Not found
2020-01-29T14:06:00.313Z cpu9:2099233)WARNING: Fil3: 8175: Found invalid object on 59c4eb22-52356382-42b5-5cb90195e9c0 expected
2020-01-29T14:06:00.313Z cpu9:2099233)Vol3: 3374: Failed to get object 28 type 3 uuid 59c4eb22-52356382-42b5-5cb90195e9c0 FD 8029c4 gen 3 :Not found
2020-01-29T14:06:00.316Z cpu14:2099233)WARNING: Fil3: 8175: Found invalid object on 569e1890-cbba4f74-9743-5cb90195cd40 expected
2020-01-29T14:06:00.316Z cpu14:2099233)Vol3: 3374: Failed to get object 28 type 3 uuid 569e1890-cbba4f74-9743-5cb90195cd40 FD 44029c4 gen 24 :Not found
2020-01-29T14:06:00.316Z cpu14:2099233)WARNING: Fil3: 8175: Found invalid object on 569e1890-cbba4f74-9743-5cb90195cd40 expected
2020-01-29T14:06:00.316Z cpu14:2099233)Vol3: 3374: Failed to get object 28 type 3 uuid 569e1890-cbba4f74-9743-5cb90195cd40 FD 44029c4 gen 24 :Not found


Errors in /var/log/storagerm.log:

2020-01-29T15:14:10Z storageRM[2099233]: Stat file: numblocks= 2048 blocksize = 1048576
2020-01-29T15:14:10Z storageRM[2099233]: Opening slot count file /vmfs/volumes//VOLUME/.naa.60002ac000000000000000370001930f/slotsfile
2020-01-29T15:14:10Z storageRM[2099233]: open /vmfs/volumes//VOLUME/.naa.60002ac000000000000000370001930f/slotsfile(0x10000042, 0x0) failed: No such file or directory
2020-01-29T15:14:10Z storageRM[2099233]: Giving UP No such file or directory Error 195887105 opening SLOT file /vmfs/volumes//VOLUME/.naa.60002ac000000000000000370001930f/slotsfile
2020-01-29T15:14:10Z storageRM[2099233]: Error -1 in opening & reading the slot file
2020-01-29T15:14:10Z storageRM[2099233]: Error in opening stat file for device: naa.60002ac000000000000000370001930f. Ignoring this device.
2020-01-29T15:14:10Z storageRM[2099233]: Stat file: numblocks= 2048 blocksize = 1048576
2020-01-29T15:14:10Z storageRM[2099233]: Opening slot count file /vmfs/volumes//VOLUME/.naa.60002ac000000000000000020001930f/slotsfile
2020-01-29T15:14:10Z storageRM[2099233]: open /vmfs/volumes//VOLUME/.naa.60002ac000000000000000020001930f/slotsfile(0x10000042, 0x0) failed: No such file or directory
2020-01-29T15:14:10Z storageRM[2099233]: Giving UP No such file or directory Error 195887105 opening SLOT file /vmfs/volumes//VOLUME/.naa.60002ac000000000000000020001930f/slotsfile


I could not see or find any problem these erros could have caused.

The solution

I opened a support ticket and after some research, support engineer found the solution to stop errors.

On one of the accessing ESXi hosts do the following:

• Find volume-name to naa-Number by running one of these commands:
  • • esxcfg-scsidevs --vmfs
    • esxcli storage vmfs extent list
• Change current folder to volume-root: cd /vmfs/volume/volume_name. When listing all files ls -alh – incl. hidden – you should see a directory named like naa-identifier:
  
• Stop the storageRM service – /etc/init.d/storageRM stop
• Rename existing folder mv .naa.60002ac000000000000000020001930f .naa.60002ac000000000000000020001930f.old
• Start the storageRM service – /etc/init.d/storageRM start
• Check file gets re-created: .naa.60002ac000000000000000020001930f/slotsfile
• Check error again: tail -f /var/log/vmkernel.log | grep -i "uuid"

There should be no more errors.