Sunday, April 13, 2014

Fire in FireEye Valuation Gets Doused (slightly) With Release of NSS Breach Study Report – He Said, She Said Begins

 NSS Labs issued their Breach Detection Security Value Map on April 2  Neither FireEye nor AhnLab can be pleased.  In brief, the Value Map  measures security effectiveness on the Y-axis and Total Cost of Ownership (TCO) per protected MbPS on the X-axis.  AhnLab and FireEye finished in the dreaded lower left hand corner with FireEye coming in last in security effectiveness (AhnLab was close).  AhnLab had the highest TCO per Protected MBPS. The other four company’s products were in the upper right hand quadrant (Quadrant 1), Fidelis, Fortinet, Trend Micro, and SourceFire. They were all around 98% to 99% effective in NSS testing.  SourceFire was the winner, overall. 

From NSS, “Quadrant 1 contains those products that are recommended for both security effectiveness/management and value.  These devices provide a very high level of protection, manageability, and value for money.”  This document is publicly available from Fortinet as is a detailed report for their FortiSandbox 200D appliance.

Key findings mentioned in the press release - “Four of Six Leading Vendors Receive Coveted NSS ‘Recommended Rating’”
  • Four of six products tested achieved over 95% in overall security effectiveness:   five of the six also received a 0% false positive rate.  AhnLab was the sixth with a 7% false positive rate.  FireEye had the lowest security effectiveness, around 94.5%. 
  • Money Doesn’t Always Buy the Best Security: Total Cost of Ownership per Protected-Mbps ranged from $231 to $468 with the highest priced solution,   Conversely, Sourcefire (Cisco) had the lowest TCO and also received one of the highest security effectiveness ratings.
  • All BDS Solutions Performed At or Above Vendor Throughput Claims

NSS Labs did not receive any compensation in return for vendor participation; All testing and research was conducted free of charge.

FireEye Stock Price (FEYE)

FireEye stock has dropped 49% percent from its March high of $97.35 to closing at $47.33 on April 11.  52-week range - $33.30 - $97.35.  It will be interesting now to see how the stock performs.  Q1 results won’t be announced until May 6.  Note -  The stock was at   $61.49 on April 2 when the report was released.  FireEye's  Q1 results won’t be comparable to    last year’s Q1 since revenue from their Mandiant acquisition after January 1 will be included.  The stock is up about 15% since the beginning of the year.  NASDAQ is down about 3% ovr the same period of time.

When you’re the market share leader, finishing low in an impartial test, one defense is to attack the attacker.
He Said - FireEye

"We are a vendor that specializes in advanced attack detection, not in detecting known, stale samples,” Gupta, FireEye Vice President of Products said.  "We ran their malware samples in our lab and detected every single one of them." A valid test would have used a zero-day exploit to evaluate the detection capabilities of the appliances or, at a minimum, the testing could have been done in a live, customer environment, Gupta added.

FireEye was quick to reply in a blog “Real World vs. Lab Testing: The FireEye Response to NSS Labs Breach Detection Systems Report” At a high level: 
  • Issue #1:  Poor sample selection
  • Issue #2:  Differing definitions of advanced malware
  • Issue #3:  Poor test methodology.   

FireEye offered several paragraphs of detail for each of the above.  It is worth reading the blog.

“The best way to evaluate FireEye is for an organization to deploy our technology in their own environment and they will understand why we are the market leader in stopping advanced attacks, “said Dave Merkel, CTO in an April 2 Network World article.

She Said – NSS Labs

NSS Labs was also quick to replay in a blog “Don't Shoot the Messenger”
Their response is also good reading as most of the response consists of   a 20-bullet point “FireEye Claim” and “NSS Response” table.

“Not everyone can end up in the top right quadrant of the NSS Labs Security Value Map™ (SVM), so it is not unusual for someone to be unhappy.  It is, however, unusual for someone to behave the way FireEye did in this instance.  Normally we would not respond to such attacks, but there are a number of untruths and misdirection’s in their blog post that we feel we must address”, stated Bob Walder, President, and Chief Research Officer at NSS.  “FireEye’s results were not that bad.  The real issue here is that FireEye now has credible competition in the BDS market place and the data from this NSS test shows it.”

How Did This Begin

Three companies were tested last summer by NSS Labs in their initial breach study, AhnLab, FireEye, and Fidelis.  Fidelis made their report publicly available and challenged FireEye to do the same.  AhnLab issued a press release about their results, and in a blog went, “FireEye, hello?”  No press release by FireEye on their results.  Demerits to publications not asking about this!  With respect to the three companies, NSS has a multi-page document letting the firms tested know what they can do with the test results.  One thing they can’t do is start-doing comparisons with other companies, combining charts, et cetera from the reports.  The reports were available for purchase.

And What about NSS Labs’ Reputation?

In “IT Security Survey 2014” by  test group AV-Comparatives (,   issued in February, NSS Labs came in ninth out of 15 vendors.  Over 5800 users responded to the survey.  

Timing Means Everything When Stock is Sold

On March 12, insider transactions of FireEye stock at $79.54 included: 
  1. Norwest Venture Partners IX, LP sold 2 million shares, grossing $160 million.
  2. FireEye CTO Aziz Ashar sold 1.04 million shares, grossing $83 million
  3. FireEye CEO Dave DeWalt sold 486 thousand shares grossing $38 million

Insiders can’t sell shares whenever they want.  There are windows near the release of financial results that they can’t do anything.  A more comprehensive list of insider transactions can be viewed at

It’s difficult to test security products.  Every environment is unique.  The best way for companies to evaluate products is to bring them in and to look at tests by reliable test groups.  The report by NSS Labs probably means   that FireEye will face more testing in house by potential vendors  rather than just be evaluated separately. 

Twitter - ckensek

No comments: