Hello,
I'm not sure how many problems are lurking in my setup but I'll try to explain what I am experiencing.
I have a few ESX hosts with several VMs on them. (I'll update this post soon with all the hardware and VM specs)
After running for days, one of the VMs all of a sudden experienced very high Avg Disk Read QL, in the thousands (~1000-4000) continuously. Total %PT is low and memory is fine. That VM was killing my application which runs across several VMs; basically others VMs access the poorly performing VM so they end up spinning waiting for data that's taking forever to be read and transfered. This is basically the problem in a nutshell.
There are two disks on the ESX host; the poor VM runs on one of them, 3 other VMs run on the second disk. The 3 VMs are fine. So I thought initially that it's a disk read head problem. I migrated the VM to another ESX Host, but the problem persisted. I then noticed that the VM was first created with Thin storage provisionning (256GB), so I migrated it once again to another ESX Host with Thick provisionning this time (256GB), but the problem still persisted.
I run windows disk Fragmentation analysis within the VM's OS and it showed 5% fragmentation.
I'm really not sure what else to look at. Please ask me questions regarding setup and I'll get the answers, I totally understand that you need to get the full picture.
Thanks in advance.