I was troubleshooting storage related problems at a customer site for some time. We run into massive ESXi storage connection problems after installing a new array. Interestingly enough problems occurred on already existing storage a few hours after new array was installed. Because searching for root cause took some time, I thought it is worth to share.
I am in the middle of a troubleshooting scenario. It is about connection problems from ESXi hosts to their storage devices. Normally everything works fine but at from time to time, there are tons of errors on HBAs in hosts. Therefor I want to watch HBA errors in ESXi hosts with PowerCLI.
These days I had to troubleshoot a problem in a vCenter environment. A few VMs shown with absolute path to vmx file in vCenter inventory. Normally this occurs when a ESXi boots with no connection to its previous mounted storage volumes. VMs located on these volumes are shown like this in inventory. But this wasn’t the case here: VMs were running and no host was […]
I want to share the solution of flooding errors I had after an vSphere upgrade. Errors are probably not connected to upgrade. The problem I got flooded by errors for VMFS datastores in log files. Errors in /var/log/vmkernel.log: 2020-01-29T14:06:00.311Z cpu8:2099233)WARNING: Fil3: 8175: Found invalid object on 59c4eb22-52356382-42b5-5cb90195e9c0 expected 2020-01-29T14:06:00.311Z cpu8:2099233)Vol3: 3374: Failed to get object 28 type 3 uuid 59c4eb22-52356382-42b5-5cb90195e9c0 FD 8029c4 gen 3 :Not […]
First, what does active/standby mean. For this blog post, it means the array hast at least 2 controllers, one presents ALL LUNs to the hosts, the second controller presents NO active LUNs path to the hosts. Just in case of a manual or automatic failover, the second controller takes-over ALL LUN-presentations – and active paths. Examples of such arrays are: [very old one] HP MSA 1000 HPE Nimble […]
These days I had a problem with full ESXi ramdisk. Running vdf -h shows tmp as 100% used. Furthermore there are a lot of errors in vmkernel.log like: VisorFSRam: 233: Cannot extend visorfs file /var/lib/vmware/hostd/journal/ because its ramdisk (root) is full. There are some other problems these hosts suffer from time to time: vSphere HA issus like master of cluster cannot be found/elected. Connection problems with […]
Recently I had to investigate an interesting behavior of ESXi hosts during Storage vMotion between two arrays of the same vendor – in this case HDS (G600 –> G700/F700). In this post I list observed symptoms and a very simple way to boost performance in this situation.
In this blog post you can see how to configure IO limit for VM at VMDK level and how ESXi hosts executes these limits. Furthermore you will see differences between SIOC (Storage IO Control) v1 and v2 and how IO size matters.
There are a few CLI methods to configure advanced settings in ESXi Hosts. It depends on where you are (vMA/Linux, Windows Client, Console) and what you want to do (configure all hosts or just a specific host, …) which option fits best. Here I describe the options: native PowerCLI, esxcli (console), esxcli (get-esxcli), esxcli (get-esxcli -v2) to: List All settings All settings in a specific […]
Some time ago differentiation between PDL (Permanent Device Loss) and APD (All Path Down) was introduced to vSphere. In a PDL situation, the hosts does not expect the device to return. PDL happens for example, when an administrator un-presents a LUN from a host. APD on the other hand is completely unplanned. By default the host tries to get the device back. Starting with vSphere […]