NSS Labs issued their Breach Detection Security Value Map on
April 2 Neither FireEye nor AhnLab can be pleased. In brief, the Value Map measures security
effectiveness on the Y-axis and Total Cost of Ownership (TCO) per protected
MbPS on the X-axis. AhnLab and FireEye finished in the
dreaded lower left hand corner with FireEye coming in last in security
effectiveness (AhnLab was close). AhnLab had the highest TCO per
Protected MBPS. The other four company’s products were in the upper right hand
quadrant (Quadrant 1), Fidelis, Fortinet, Trend Micro, and SourceFire. They were all around 98% to 99% effective in
NSS testing. SourceFire was the winner, overall.
From NSS, “Quadrant 1 contains those products
that are recommended for both security effectiveness/management and value. These devices provide a very high level of protection,
manageability, and value for money.”
This document is publicly available from Fortinet as is a detailed
report for their FortiSandbox 200D appliance.
Key findings mentioned in the press release - “Four of Six
Leading Vendors Receive Coveted NSS ‘Recommended Rating’”
- Four of six products tested achieved over 95% in overall security effectiveness: five of the six also received a 0% false positive rate. AhnLab was the sixth with a 7% false positive rate. FireEye had the lowest security effectiveness, around 94.5%.
- Money Doesn’t Always Buy the Best Security: Total Cost of Ownership per Protected-Mbps ranged from $231 to $468 with the highest priced solution, Conversely, Sourcefire (Cisco) had the lowest TCO and also received one of the highest security effectiveness ratings.
- All BDS Solutions Performed At or Above Vendor Throughput Claims
NSS Labs did not receive any compensation in return for
vendor participation; All testing and research was conducted free of charge.
FireEye Stock Price (FEYE)
FireEye stock has dropped 49% percent from its March high of $97.35 to closing at $47.33 on April 11. 52-week range - $33.30 - $97.35. It will be interesting now to see how the stock performs. Q1 results won’t be announced until May 6. Note - The stock was at $61.49 on April 2 when the report was released. FireEye's Q1 results won’t be comparable to last year’s Q1 since revenue from their Mandiant acquisition after January 1 will be included. The stock is up about 15% since the beginning of the year. NASDAQ is down about 3% ovr the same period of time.
When you’re the market share leader, finishing low in an
impartial test, one defense is to attack the attacker.
He Said - FireEye
"We are a vendor that specializes in advanced attack
detection, not in detecting known, stale samples,” Gupta, FireEye Vice
President of Products said. "We ran
their malware samples in our lab and detected every single one of them." A valid test would have used a zero-day exploit to evaluate
the detection capabilities of the appliances or, at a minimum, the testing
could have been done in a live, customer environment, Gupta added.
FireEye was quick to reply in a blog “Real World vs. Lab
Testing: The FireEye Response to NSS Labs Breach Detection Systems Report” At a
high level:
- Issue #1: Poor sample selection
- Issue #2: Differing definitions of advanced malware
- Issue #3: Poor test methodology.
FireEye offered several paragraphs of detail for each of the
above. It is worth reading the blog.
“The best way to evaluate FireEye is for an organization to
deploy our technology in their own environment and they will understand why we
are the market leader in stopping advanced attacks, “said Dave Merkel, CTO in
an April 2 Network World article.
She Said – NSS Labs
NSS Labs was also quick to replay in a blog “Don't Shoot the
Messenger”
Their response is also good reading as most of the response
consists of a 20-bullet point “FireEye Claim” and “NSS
Response” table.
“Not everyone can end up in the top right quadrant of the
NSS Labs Security Value Map™ (SVM), so it is not unusual for someone to be
unhappy. It is, however, unusual for
someone to behave the way FireEye did in this instance. Normally we would not respond to such
attacks, but there are a number of untruths and misdirection’s in their blog
post that we feel we must address”, stated Bob Walder, President, and Chief
Research Officer at NSS. “FireEye’s
results were not that bad. The real
issue here is that FireEye now has credible competition in the BDS market place
and the data from this NSS test shows it.”
How Did This Begin
Three companies were tested last summer by NSS Labs in their
initial breach study, AhnLab, FireEye, and Fidelis. Fidelis made their report publicly available
and challenged FireEye to do the same. AhnLab
issued a press release about their results, and in a blog went, “FireEye,
hello?” No press release by FireEye on
their results. Demerits to publications
not asking about this! With respect to
the three companies, NSS has a multi-page document letting the firms tested know
what they can do with the test results. One
thing they can’t do is start-doing comparisons with other companies, combining
charts, et cetera from the reports. The
reports were available for purchase.
And What about NSS Labs’ Reputation?
In “IT Security Survey 2014” by test group AV-Comparatives (www.av-comparatves.org), issued
in February, NSS Labs came in ninth out of 15 vendors. Over 5800 users responded to the survey.
Timing Means Everything When Stock is Sold
On March 12, insider transactions of FireEye stock at $79.54
included:
- Norwest Venture Partners IX, LP sold 2 million shares, grossing $160 million.
- FireEye CTO Aziz Ashar sold 1.04 million shares, grossing $83 million
- FireEye CEO Dave DeWalt sold 486 thousand shares grossing $38 million
Insiders can’t sell shares whenever they want. There are windows near the release of financial results that they can’t do anything. A more comprehensive list of insider transactions can be
viewed at
It’s difficult to test security products. Every environment is unique. The best way for companies to evaluate
products is to bring them in and to look at tests by reliable test groups. The report by NSS Labs probably means that
FireEye will face more testing in house by potential vendors rather than just be evaluated separately.
Twitter - ckensek
Posts
