Most VMware admins are familiar with VAAI (vStorage API for Array Integration), a key API to make storage faster, more efficient, and easier to manage in vSphere environments. In this blog, I explain how VAAI affects storage within vSphere environments and what each primitive does and its inherent benefit. I also describe how Tegile flash arrays are certified by VMware for VAAI.
The below table lists VAAI primitives, their brief descriptions, and their benefit:
For more detailed descriptions, please refer to this VMware KB article #1021976
Storage Certification: is required by VMware to ensure these APIs are working correctly with the storage array, and recertification is required every time a vendor releases a major firmware update. Here’s a link to the searchable page to the VMware Compatibility Matrix, which specifies storage models, protocols, and the VAAI primitives that are officially supported. Tegile supports all the VAAI primitives for NFS, FC and iSCSI protocols.
VMware Compatibility Matrix: VMware publishes the compatibility matrix based on submitted and accepted certifications from hardware vendors. As an example, the following screenshot shows a partial listing for a Tegile IntelliFlash™ T3800 array as shown in the VMware compatibility matrix for search criteria: Partner Name: Tegile Systems and Array Type=FC.
The following screenshot (based on a search for Tegile and FC) shows how the VMware compatibility list shows the VAAI block primitives certified for Tegile T3800 array with IntelliFlash 3.0 firmware.
Tegile LUNS and VAAI: To check which VAAI primitives are supported from the command-line, use the following command after logging in to an ESX host. Ironically, the naming for each of these is different in different places, command outputs, compatibility list, and documentation.
Displaying VAAI Extended LUN Statistics
It is useful to monitor VAAI operations if you encounter any performance issues performing certain operations. For example, you may want to monitor VAAI operations if it’s taking a long time to clone or run SCSI-UNMAP. In fact, it is always recommended to monitor SCSI-UNMAP when reclaiming space.
VMware vSphere shell has a performance-monitoring tool called ESXTOP. ESXTOP has the ability to display extended statistics for LUN(s), which can be selectively enabled to monitor if a particular VAAI counter is incremented or not on a VAAI operation. The following sections show how extended statistics for VAAI performance can be monitored using ESXTOP.
Monitoring VAAI primitive performance:
- Start ESXTOP on the ESXi host
- Press “u” on the keyboard to look at Disk VAAI statistics (without extended stats)
- Press “f” to add new fields as below and press “O” and “P” to enable them. You could disable others if you only want VAAI statistics. Pressing a parameter again will disable it (toggle)
(partial listing – * means that column is displayed in ESXtop)
* O: VAAISTATS= VAAI Stats
* P: VAAILATSTATS/cmd = VAAI Latency Stats (ms)
- Press any key to return to main (disk) screen. You will see additional fields as below for VAAI Stats.
CLONE_RD – number of successfully completed CLONE commands where disk device was a source
CLONE_WR – number of successfully completed CLONE commands where disk device was a destination
CLONE_F – number of failed CLONE commands
MBC_RD/s – CLONE data read per second (in MB)
MBC_WR/s – CLONE data written per second (in MB)
ATS Operations (atomic test & copy):
ATS – number of successful ATS operations
ATSF – number of failed ATS operations
Block Zero Operations:
ZERO – number of successful block zero operations
ZERO-F – number of failed block zero operations
MBZero/s – Megabytes reclaimed
DELETE – no of successful delete operations
DELETE-F – no of failed delete operations
MBDEL/S – Megabytes deleted
- Either the array doesn’t no support the VAAI primitive; in this case it is better to disable the primitive for better performance;
- Or it’s a bug with the VAAI implementation; if that’s the case, contact your storage vendor to get this issue fixed via firmware patch and then enable it.
Keep in mind though, when a VAAI primitive is disabled, it gets disabled for the entire host that is connected to the SAN and all storage enclosure. Therefore, selectively disabling a VAAI primitive by array vendor is not an option. There is a VMware KB article #1033665 that shows you how to disable one or more VAAI primitives if required.
As of IntelliFlash version 3.5, Tegile supports VAAI for NAS for the following additional primitives:
- Full Copy
- NFS Fast File Clone (will enable VCAI [View Composer Array Integration] for linked clones)
For more info on VAAI for NAS, refer to the following document for more details
A storage array that supports VMware VAAI primitives provides significant advantages in performance, manageability and efficiency. VMware’s certification program ensures that customers can reap these benefits when they’re using VMware-certified storage.
VMware storage compatibility matrix is an easy and searchable page that provides confirmation of certified storage with vSphere to a customer before purchasing and deploying it.
It is also possible to determine if VAAI primitives are supported from ESXI shell command-line.
ESXtop, a native ESXi performance utility provides an easy way to monitor and measure VAAI performance. VAAI primitives can be disabled as a stop-gap solution, if there are too many failed VAAI operations that are causing performance issues.