Improve iSCSI latency in Windows

Recently we had to troubleshoot rather high storage latency at a customer site. The environment consists of Hyper-V hosts connecting to 3PAR arrays via iSCSI. 3PAR arrays are all-flash arrays (AFA). I know such problems in vSphere environments too. There higher storage latency can be caused by delayed acknowledgement (DelAck). Read more about this issue and how to configure DelAck on vSphere ESXi hosts here. Because of this, DelAck was the first guess here too. Read here how to configure delayed acknowledgement and improve iSCSI latency in Windows.

First of all: When your host generates a lot of IOs, DelAck in Windows will not be a problem. You will see increased latency when your host operates at a level of up to 200 – 300 IOs/sec (IOps). I do not have an exact threshold. It depends on the array, IO-size, disks, network, … DelAck causes higher latency, because it generates additional time by waiting for network packets – like IOs. So with few IOs, Windows waits for more IOs to acknowledge them before processing them.

You can configure DelAck with two parameters in Windows. First is the number of packets after that an acknowledgement is sent. Second is the timeout to wait for the rest of packets.

Configuration is done with PowerShell. You need to configure TCP-settings and Transport-Filter. Latter is for applying TCP-settings to filtered TCP-connections.

Improve iSCSI latency in Windows by configure delayed acknowledgement

TCP-settings are stored as a kind of profile. You can view configured settings by running:

Get-NetTCPSetting

You will see at least this profiles:

• Automatic
• InternetCustom
• DatacenterCustom
• Compat
• Datacenter
• Internet

Initially just *Custom-profiles where able to be changed. Since Windows 2012, Datacenter and Internet can be changed as well. Note: Windows 10 does not support to change any of them!

To check current applied profile, run this command:

Get-NetTCPConnection | Where-Object {$_.RemoteAddress -like "10.10.10.*"}  | Select-Object LocalAddress, LocalPort, RemoteAddress, RemotePort, AppliedSetting

In my case “10.10.10.*” points to iSCSI network IP range. Column AppliedSetting shows one of the settings above. This was strange in my case, because I read Internet there. I recommend to configure DatacenterCustom for iSCSI connections. We set the number of packets to wait for to 1. Default value is 2. Because we do not wait for further packets, wait timeout (DelayedAckTimeout(ms)) is not applicable.

Set-NetTCPSetting -SettingName DatacenterCustom -DelayedAckFrequency 1

Next we configure filter to apply DatacenterCustom to our iSCSI connections:

New-NetTransportFilter -SettingName DatacenterCustom -DestinationPrefix 10.10.10.0/24

Note notation of network range. You can also use different filter options like target port. But to do so you need to configure local port as well. More details here.

To apply settings to connections a reboot is required. Check settings afterwards.

Result in my case: latency halves.