3PAR (iSCSI) – resolve high write latency on ESXi hosts
FC AFA (all flash array) 3PAR systems show quite a good latency on reads and writes. When operating an iSCSI AFA 3PAR it could happen that the systems show a rather high write latency on ESXi hosts. In this post you can read how to fix this.
To show write (and read) latency (service time) on virtual LUNs on 3PAR run:
statvlun -ni -rw. On ESXi hosts device latency is shown in
DAVG/cmd. For AFA 3PAR in normal circumstances, latency about 1ms is quite OK. In an iSCSI system connected to ESXi hosts I had recently write latency average about 4ms, which is too high.
The solution for this situation is simple. Just disable delayed acknowledgement (DelAck) on iSCSI targets in ESXi hosts. DelAck is an TCP concept to handle network congestion. To show already occurred DelAcks in a 3PAR system, run:
statcmp -iter 1. By the way, it is also good practice to disable DelAck when ESXi host is connected to LeftHand targets.
To disable DelAck on ESXi host:
- Set host in maintenance mode.
- Go to properties of software iSCSI adapter.
- Edit all 3PAR target on Dynamic Discovery tab.
- In Advanced go to option DelayedAck (end of the list).
- Clear Inherit from parent and Enabled.
- reboot host.
[update] Here you can find how to disable delayed acknowledgement by using PowerCLI.
[update] because this is important for Windows as well, check this post for configuring DelAck in Windows.
Here is a service time graph of a time interval of rebooting connected hosts after disabling DelAck:
Blue line shows write latency, red line is for read latency; green for average of both.