We moved our search into the vSphere console to see if there was anything happening at the virtualized hardware level. By using the “Past 24 Hours” functionality of the performance tab. We were able to “See” the performance degradation, something was causing a significant increase in the number of read requests to disk. We checked again the following day, and saw the exact same issue. Now, that we had a definitive time range, we began reviewing what was happening at night during that period of time. What we found was that it was one of our nightly interfaces for transferring data between our application and our ERP software. 

Next we dove into what was causing the issue. after a bit of research, we found the suspect select statement. Further analysis, led us to find that the table being selected from did not have the correct indexing configured. We configured the indexes, and let the nightly cycle run again. Success!! Happy end users!

This nightly issue was getting progressively worse from the day that the interface was implemented. When first implemented, there were minimal numbers of rows in the table, so the indexing need was minimal. but as time went on, and the number of rows increased, the select statement became more costly in disk access.  

So lessons learned. Something as simple as a small interface to transfer data between one application to another, can have long term implications.

It appears that while Mcafee integrated the patch 6 binaries into the VSE package, they did not update any version numbers. The client and EPO service does not know that it is a newer version. And they get confused and just quit playing well together.