Watch HBA errors in ESXi hosts with PowerCLI

I am in the middle of a troubleshooting scenario. It is about connection problems from ESXi hosts to their storage devices. Normally everything works fine but at from time to time, there are tons of errors on HBAs in hosts. Therefor I want to watch HBA errors in ESXi hosts with PowerCLI.

The error counter can be show by running the following command in ESXi console:
esxcli storage core adapter stats get
Because it is not very handy to manually run this command on every host, I wrote a short PowerCLI script to do this for every (selected) host. The following script repeats this – by default – every two seconds. So it is like the watch command in Linux

# Time to wait to next refresh
$Seconds = 2
# Select hosts
$VMHosts = Get-VMHost

$Result = @()
[array]$esxcli = @()
$i = 0

foreach ($VMHost in $VMHosts) {

    $HBAErrors = [PScustomObject] @{
        Time = Get-Date -UFormat '%H:%M:%S'
        HostName = ''
        HBA1 = ''
        HBA1_Absolute = 0
        HBA1_Delta = 0
        HBA2 = ''
        HBA2_Absolute = 0
        HBA2_Delta = 0
    }
    $HBAs = Get-VMHost $VMHost | Get-VMHostHba -Type FibreChannel
    $esxcli += $vMHost | Get-EsxCli 
    $HBAErrors.HostName = $VMHost.Name
    $HBAErrors.HBA1 = $HBAs.Device[0]
    [int]$HBAErrors.HBA1_Absolute = $esxcli[$i].storage.core.adapter.stats.get($HBAErrors.HBA1).FailedCommands
    $HBAErrors.HBA2 = $HBAs.Device[1]
    [int]$HBAErrors.HBA2_Absolute = $esxcli[$i].storage.core.adapter.stats.get($HBAErrors.HBA2).FailedCommands

    $Result += $HBAErrors
    $i++
}

$Result | ft -AutoSize
while (1 -eq 1) {
    Start-Sleep -Seconds $Seconds
    for ($i = 0; $i -le ($Result.Length -1); $i++) {
        $Result[$i].Time = Get-Date -UFormat '%H:%M:%S'

        $temp = $esxcli[$i].storage.core.adapter.stats.get($Result[$i].HBA1).FailedCommands
        [int]$Result[$i].HBA1_Delta = $temp - $Result[$i].HBA1_Absolute 
        [int]$Result[$i].HBA1_Absolute = $temp

        $temp = $esxcli[$i].storage.core.adapter.stats.get($Result[$i].HBA2).FailedCommands
        [int]$Result[$i].HBA2_Delta = $temp - $Result[$i].HBA2_Absolute 
        [int]$Result[$i].HBA2_Absolute = $temp
    }
    $Result | ft -AutoSize 
}

Script notes

The script is very simple and should be easy to read. Because of this it should also be easy to customize. The following notes describe the default behavior.

• Script can be stopped by pressing Ctrl+c.
• Only Fibre Channel HBAs are shown. This can be changed in line 22.
• Only two HBAs are selected.
• Seconds to wait for the next run can be customized at the beginning of the script, by changing the variable $seconds.
• Select the hosts you want to show errors for by changing variable $VMHots.
• Error counters are shown as absolute value and delta value. Later shows errors since last run.
• This error counter is just one of many that can be listed by esxcli. See here more about storage troubleshooting. esxcli command here can easily be changed to show other errors as well.

I hope this script to watch HBA errors in ESXi hosts with PowerCLI is useful for others too! Feedback is very welcome!