AhnLab Raises Issues with Recent NSS Labs Breach Detection Study

FireEye isn’t the only vendor displeased with their results and NSS Labs' methodology for their  latest Security Value Map.  AhnLab, whose Malware Defense System (MDS) product finished near the bottom of the Breach Detection Systems Security Value Map adjacent to FireEye, has posted their displeasure with the testing on their home page. 

AhnLab declined to participate in the 2014 public test. AhnLab, Fidelis, and FireEye had participated in the 2013 private test. Ultimately, Fidelis made their results publicly available on their website. Neither FireEye nor AhnLab chose to do so, though AhnLab did release some of the Malware Defense System results.

NSS Labs’  test evaluated 6 products from leading BDS vendors.  Four of the six products received the   "Recommended" rating from NSS, Sourcefire, Trend Micro, Fortinet, and Fidelis.  Neither FireEye nor AhnLab didn’t.   

AhnLab’s  Main Points

1. Two separate public tests, were consolidated into one report without notice -   AhnLab wrote that NSS never informed them the results would be published regardless of participation. This may or may not be true as many of the participants on the AhnLab side are no longer with the organization.
2. Two separate tests from two different years require two separate reports -  If the same malware sample set was used from 2013 for the 2014 test, AhnLab felt that it  would be inaccurate to publish all of the participants, from 2013 and 2014 together,  because newcomers to the study may have (had) a time advantage.

For complete details, go to http://us.ahnlab.com/html/notice/20140408.jsp

For a copy of the NSS Labs April Breach Detection Systems Security Value Map (SVM) and Comparative Analysis Reports (CARs), go to https://www.nsslabs.com/breach-detection-systems-bds-security-value-map-download

Some of the above sounds like a failure to communicate on both NSS Labs and AhnLab’s part. Neither side appears to have done due diligence here.

Only three companies completed participation in the 2013 test, not ten or more, as AhnLab writes in their response.  They may have a valid response about products with several more months “experience” having their results compared to products without that experience.   That notwithstanding, 3^rd party test results is one aspect of comparing products that companies need to utilize. The test results demonstrate that there is more than just FireEye, Fidelis, and AhnLab that need to be considered.

When Being an “A” Company Rates a “D”. AV-Comparatives Releases “File Detection Test of Malicious Software” Report

Not a stellar performance by three firms beginning with A in AV-Comparatives March “File Detection Test of Malicious Software.”  Avast – 20th, missing 2.3% of the samples.  AVG Technologies – 21^st, misses 2.5% of the samples, and AhnLab 22^nd, missing 11% of the samples.  Baidu broke the A’s stranglehold on the bottom by leading all companies with 111 false positives, followed by Avast with 95.

Sixteen products did receive AV-Comparatives’ three star designation, led by Kaspersky, F-Secure, and eScan, respectively.  Avira, another A company, also received three stars. 

AV-Comparatives takes care to point out that for this test, “Although very important, the file detection rate of a product is only one aspect of a complete anti-virus product.  AV-Comparatives also   provides a whole-product dynamic “real-world” protection test, as well as other test reports that cover different aspects/features of the products”.

 

The “Whole Product Dynamic Real World Detection" and “File Detection Test of Malicious Software”  tests   are both   available on the AV-Comparatives web site, www.av-comparatives.org

One interesting thing about the products tested in this report is that nine of the engines under the hood in testing were licensed from two companies, BitDefender and Avira.  Details are available in the report.  BitDefender  has an overall detection rate of 99.5%.

About AV-Comparatives

AV-Comparatives is an independent not-for-profit organization offering systematic testing that checks whether security software, such as PC/Mac-based antivirus products and mobile security solutions, lives up to its promises.  Using one of the largest sample collections worldwide, it creates a real-world environment for truly accurate testing. 

Fire in FireEye Valuation Gets Doused (slightly) With Release of NSS Breach Study Report – He Said, She Said Begins

 NSS Labs issued their Breach Detection Security Value Map on April 2  Neither FireEye nor AhnLab can be pleased.  In brief, the Value Map  measures security effectiveness on the Y-axis and Total Cost of Ownership (TCO) per protected MbPS on the X-axis.  AhnLab and FireEye finished in the dreaded lower left hand corner with FireEye coming in last in security effectiveness (AhnLab was close).  AhnLab had the highest TCO per Protected MBPS. The other four company’s products were in the upper right hand quadrant (Quadrant 1), Fidelis, Fortinet, Trend Micro, and SourceFire. They were all around 98% to 99% effective in NSS testing.  SourceFire was the winner, overall. 

From NSS, “Quadrant 1 contains those products that are recommended for both security effectiveness/management and value.  These devices provide a very high level of protection, manageability, and value for money.”  This document is publicly available from Fortinet as is a detailed report for their FortiSandbox 200D appliance.

http://www.fortinet.com/resource_center/whitepapers/breach-detection-systems-beyond-hype.html

Key findings mentioned in the press release - “Four of Six Leading Vendors Receive Coveted NSS ‘Recommended Rating’”

• Four of six products tested achieved over 95% in overall security effectiveness:   five of the six also received a 0% false positive rate.  AhnLab was the sixth with a 7% false positive rate.  FireEye had the lowest security effectiveness, around 94.5%. 
• Money Doesn’t Always Buy the Best Security: Total Cost of Ownership per Protected-Mbps ranged from $231 to $468 with the highest priced solution,   Conversely, Sourcefire (Cisco) had the lowest TCO and also received one of the highest security effectiveness ratings.
• All BDS Solutions Performed At or Above Vendor Throughput Claims

https://www.nsslabs.com/news/press-releases/nss-labs-reveals-first-security-value-map%E2%84%A2-breach-detection-systems

NSS Labs did not receive any compensation in return for vendor participation; All testing and research was conducted free of charge.

FireEye Stock Price (FEYE)

FireEye stock has dropped 49% percent from its March high of $97.35 to closing at $47.33 on April 11.  52-week range - $33.30 - $97.35.  It will be interesting now to see how the stock performs.  Q1 results won’t be announced until May 6.  Note -  The stock was at   $61.49 on April 2 when the report was released.  FireEye's  Q1 results won’t be comparable to    last year’s Q1 since revenue from their Mandiant acquisition after January 1 will be included.  The stock is up about 15% since the beginning of the year.  NASDAQ is down about 3% ovr the same period of time.

When you’re the market share leader, finishing low in an impartial test, one defense is to attack the attacker.

  

He Said - FireEye

"We are a vendor that specializes in advanced attack detection, not in detecting known, stale samples,” Gupta, FireEye Vice President of Products said.  "We ran their malware samples in our lab and detected every single one of them." A valid test would have used a zero-day exploit to evaluate the detection capabilities of the appliances or, at a minimum, the testing could have been done in a live, customer environment, Gupta added.

FireEye was quick to reply in a blog “Real World vs. Lab Testing: The FireEye Response to NSS Labs Breach Detection Systems Report” At a high level: 

• Issue #1:  Poor sample selection
• Issue #2:  Differing definitions of advanced malware
• Issue #3:  Poor test methodology.   

FireEye offered several paragraphs of detail for each of the above.  It is worth reading the blog.

http://www.fireeye.com/blog/corporate/2014/04/real-world-vs-lab-testing-the-fireeye-response-to-nss-labs-breach-detection-systems-report.html

“The best way to evaluate FireEye is for an organization to deploy our technology in their own environment and they will understand why we are the market leader in stopping advanced attacks, “said Dave Merkel, CTO in an April 2 Network World article.

She Said – NSS Labs

NSS Labs was also quick to replay in a blog “Don't Shoot the Messenger”

Their response is also good reading as most of the response consists of   a 20-bullet point “FireEye Claim” and “NSS Response” table.

“Not everyone can end up in the top right quadrant of the NSS Labs Security Value Map™ (SVM), so it is not unusual for someone to be unhappy.  It is, however, unusual for someone to behave the way FireEye did in this instance.  Normally we would not respond to such attacks, but there are a number of untruths and misdirection’s in their blog post that we feel we must address”, stated Bob Walder, President, and Chief Research Officer at NSS.  “FireEye’s results were not that bad.  The real issue here is that FireEye now has credible competition in the BDS market place and the data from this NSS test shows it.”

https://www.nsslabs.com/blog/dont-shoot-messenger

How Did This Begin

Three companies were tested last summer by NSS Labs in their initial breach study, AhnLab, FireEye, and Fidelis.  Fidelis made their report publicly available and challenged FireEye to do the same.  AhnLab issued a press release about their results, and in a blog went, “FireEye, hello?”  No press release by FireEye on their results.  Demerits to publications not asking about this!  With respect to the three companies, NSS has a multi-page document letting the firms tested know what they can do with the test results.  One thing they can’t do is start-doing comparisons with other companies, combining charts, et cetera from the reports.  The reports were available for purchase.

And What about NSS Labs’ Reputation?

In “IT Security Survey 2014” by  test group AV-Comparatives (www.av-comparatves.org),   issued in February, NSS Labs came in ninth out of 15 vendors.  Over 5800 users responded to the survey.  

 http://www.av-comparatives.org/security-usage-surveys/   

Timing Means Everything When Stock is Sold

On March 12, insider transactions of FireEye stock at $79.54 included: 

1. Norwest Venture Partners IX, LP sold 2 million shares, grossing $160 million.
2. FireEye CTO Aziz Ashar sold 1.04 million shares, grossing $83 million
3. FireEye CEO Dave DeWalt sold 486 thousand shares grossing $38 million

Insiders can’t sell shares whenever they want.  There are windows near the release of financial results that they can’t do anything.  A more comprehensive list of insider transactions can be viewed at

 http://finance.yahoo.com/q/it?s=FEYE+Insider+Transactions

  

It’s difficult to test security products.  Every environment is unique.  The best way for companies to evaluate products is to bring them in and to look at tests by reliable test groups.  The report by NSS Labs probably means   that FireEye will face more testing in house by potential vendors  rather than just be evaluated separately. 

Twitter - ckensek