AhnLab Raises Issues with Recent NSS Labs Breach Detection Study

FireEye isn’t the only vendor displeased with their results and NSS Labs' methodology for their  latest Security Value Map.  AhnLab, whose Malware Defense System (MDS) product finished near the bottom of the Breach Detection Systems Security Value Map adjacent to FireEye, has posted their displeasure with the testing on their home page. 

AhnLab declined to participate in the 2014 public test. AhnLab, Fidelis, and FireEye had participated in the 2013 private test. Ultimately, Fidelis made their results publicly available on their website. Neither FireEye nor AhnLab chose to do so, though AhnLab did release some of the Malware Defense System results.

NSS Labs’  test evaluated 6 products from leading BDS vendors.  Four of the six products received the   "Recommended" rating from NSS, Sourcefire, Trend Micro, Fortinet, and Fidelis.  Neither FireEye nor AhnLab didn’t.   

AhnLab’s  Main Points

1. Two separate public tests, were consolidated into one report without notice -   AhnLab wrote that NSS never informed them the results would be published regardless of participation. This may or may not be true as many of the participants on the AhnLab side are no longer with the organization.
2. Two separate tests from two different years require two separate reports -  If the same malware sample set was used from 2013 for the 2014 test, AhnLab felt that it  would be inaccurate to publish all of the participants, from 2013 and 2014 together,  because newcomers to the study may have (had) a time advantage.

For complete details, go to http://us.ahnlab.com/html/notice/20140408.jsp

For a copy of the NSS Labs April Breach Detection Systems Security Value Map (SVM) and Comparative Analysis Reports (CARs), go to https://www.nsslabs.com/breach-detection-systems-bds-security-value-map-download

Some of the above sounds like a failure to communicate on both NSS Labs and AhnLab’s part. Neither side appears to have done due diligence here.

Only three companies completed participation in the 2013 test, not ten or more, as AhnLab writes in their response.  They may have a valid response about products with several more months “experience” having their results compared to products without that experience.   That notwithstanding, 3^rd party test results is one aspect of comparing products that companies need to utilize. The test results demonstrate that there is more than just FireEye, Fidelis, and AhnLab that need to be considered.