AV-Comparatives Real World Protection Test – August to November 2015

Kaspersky and BitDefender topped twenty companies in AV-Comparatives Real World Protection Test, August to November 2015. These companies finished in the top two, receiving three stars. Both had only one compromised file. Six other companies received three stars over the test period.  Eight of the twenty companies in the test received two stars. Default settings were used for all products.

Trivia question – which North America based malware company received three stars?

Four companies merit the Hall of Shame award for the period, garnering one or zero stars. From the bottom up, ThreatTrack Vipre, Lavasoft, Quick Heat and BullGuard.   Banished to a timeout corner   for being in triple digits for wrongly blocked files – Mcafee, ThreatTrack, and Lavasoft.

An informative graphic in the report depicts the range of protection over the four-month period for each product. The top products were extremely consistent, which is what you would want in a security solution.  The bottom products, less so. 

Not all results are being provided because there is no charge for the report. It can be downloaded at http://www.av-comparatives.org/dynamic-tests/. You can also learn more about the test methodology in the fourteen-page report.  The products tested ranged from free antivirus to internet security suites.  Kudos to AV-Comparatives for detailing some of the statistics methodologies used in compiling their report. Your eyes won’t glaze over as you read about this.

As always, the top products may not be top in terms of number of “likes” they’ve received on their respective Facebook pages.  In  the denouement, should one give more weight to independent third party testing, or a fan club?

An interesting article to read by Adam Winn at San Francisco based OPSWAT,  (www.opswat.com) Sorry Symantec - Antivirus is Not Dead .  Today’s antivirus/malware protection utilizes more than just pattern files and heuristics.

Hall of Shame and timeout corners are not part of AV-Comparatives’ formal designations. You can learn about the organization at www.av-comparatives.org

The trivia question answer – none.  McAfee and Fortinet received two stars.

Cyphort vs. FireEye – FireWhy? The Breach Detection, Advanced Persistent Threat Battle

Cyphort is taking a different tact versus the others in the breach detection, Advanced Persistent Threat (APT) market with their Cyphort Advanced Threat Protection solution (claim: complete 360º APT defense!)   Cyphort positions the company as both superior to FireEye and  able to  coexist with FireEye. Getting their nose under the tent for when renewals coming up? Shortening the review cycle when renewals come up?  Coverage for areas of a company where there aren’t FireEye appliances?  Cyphort didn’t participate in the NSS Labs Breach Detection study.

    

 FireEye is the 800-pound gorilla with respect to market revenue and visibility.  The David vs. Goliath analogy won’t work since FireEye’s CEO’s first name is Dave! Cyphort’s 2014 revenue was around $14 million. FireEye’s was $426 million (this includes revenue from the Mandiant acquisition).

Cyphort claims that their solution delivers malware lateral movement detection. They define this as "the ability to combine advanced targeted attacks and Advanced Persistent Threats (APT) detection with lateral movement." They say that their product provides a  picture of the attack as it happens and the potential spread within an organization, in real-time.

The Cyphort solution is delivered as software that can be installed on general-purpose hardware, virtual machines and cloud environments. The solution consists of four core components:

Collector:  Software-based probes deployed at strategic network locations (Internet egress points, data centers, etc.)   to collect suspect objects and communication.

Core:   This is the centralized detection component of Cyphort’s solution; Cyphort Core analyzes the collected suspicious network objects and associated metadata from the Collectors 

Manager: This is a  web-based,  administrative Interface.  It enables someone to manage the distributed deployment and provides access to reports

Threat Network: This cloud service feeds global threat intelligence to the Cyphort Core for enhanced detection of current threats. It aggregates threat information across all Cyphort installations

At RSA earlier this year,  Cyphort's co-founder and Chief Strategy Officer Fengmin Gong  said, "Today, solutions must look at every stage of the cyber kill chain."  

It’s always good to have more competition. Based on press, one would think that the APT market is the exclusive domain FireEye and the other seven companies that are part of the most recent NSS Breach Detection Systems (BDS) test!

 Is FireEye Cyphort’s Friend or Foe?

On the Cyphort site at http://www.cyphort.com/products/firewhy/   there are pair of threads prospects can go down.  

For those who already have FireEye, Cyphort claims that their Cyphort Advanced Threat Protection solution can be used to address gaps in the FireEye solution.  Their pitch is that they enhance protection.

Enterprise-wide Coverage: Unprotected sites and data centers can be covered with a single global license

Enterprise-wide Deployment: Deployment in days using the virtual machine approach

No appliance proliferation:  Cyphort claims that they cover & correlate email/web/file traffic across multiple operating systems, all in one solution

The second thread is for those considering FireEye.  Cyphort claims that they are   “the clear alternative”.  They have a nice (of course, it’s selective) grid containing points of differentiation (FireEye in ( ) :

Detection: Sandbox evasion detection, Data exfiltration detection, Multi-part threat detection, Golden image sandbox for contextual detection.  (no for all four )

Coverage: Distributed/Decoupled Design for Global Deployment using collectors (Monolithic) , Hardware/Software/VM deployment  (hardware only), Integrated Web/Email threat detection for Windows and Mac OSX threats (multiple appliances needed)

Action: Risk-based Threat Prioritization , Containment Using Existing Firewall, Web Gateway and IPS Devices, Endpoint Infection Verification (no for all three)

Scale and Flexibility: Scalability, clustered design to support any load (limited by highest appliance capacity for FireEye );  IT ecosystem Integration, open API (limited); Licensing is enterprise wide by bandwidth (per appliance for FireEye)

The Radicati Group has a  APT market share and  2015-2019 APT forecast  report available for purchase ($3000)  Radicati APT-Protection-Market-2015-2019-Brochure.pdf

For those wanting another company’s view of Cyphort’s and FireEye’s offerings, LastLine has performed  their own analyses:

Lastline vs. Cyphort

Lastline vs. FireEye

Products in the Breach Detection Systems (BDS) Security Value Map™ 2015

In the August NSS Breach Detection Systems Test,  Cisco had the highest detection rate, Blue Coat the lowest TCO.  FireEye - lower left in the grid.  As mentioned earlier, Cyphort was not in this study.

Five of the eight received a recommended rating (Those on the upper right corner of the value map). Some of the companies tested have the individual reports available on their web site.  To purchase reports, see below.  For the BDS Security Value Map Graphic:

• bds-security-value-map-graphic

Participants in the NSS Breach Detection Systems  Study:

• Blue Coat Security Analytics and Blue Coat Malware Analysis Appliance
• CheckPoint 13500 Next Generation Threat Prevention Appliance with Threat Emulation Cloud Service
• Cisco Advanced Malware Protection
• Fidelis XPS Direct 1000 & Fidelis XPS Internal 1000
• FireEye EX-3400 & NX-4400
• Fortinet FortiSandbox-1000D
• Lastline Breach  Detection Platform
• Trend Micro Deep Discovery Inspector 

 Studies are available on the NSS site. Some are available for free on the participant's site.

Palo Alto Networks, Check Point top Products in Gartner Magic Quadrant for Enterprise Network Firewalls - 2014

As is probably no big surprise to those in the industry and those purchasing network security products, Palo Alto Networks (PAN) and Check Point had the top rated products in the 2014 Gartner Magic Quadrant for Enterprise Network Firewalls.  The report came out in April.  These are the only two companies in the Leaders Quadrant, with Palo Alto Networks leading on Completeness of Vision and Check Point for Ability to Execute. Fortinet and Cisco were the closest to the in the Challengers quadrant.   The report, ID:G00258296 is available on the PAN web site for those who register. http://connect.paloaltonetworks.com/gartner-mq-2014

Palo Alto Networks pretty much was the originator of the acronym NGFW or Next Generation Firewall, and PAN and Check Point Software Technologies companies compete for many of the same customers. Last year, PAN   introduced their Wildfire infrastructure, enabling the PAN firewall to detect and stop Advanced Persistent Threats (APTs) This is offered to customers via the public cloud or can be deployed as a private cloud.   Gartner also wrote that PAN    was consistently on most NGFW competitive shortlists.  PANS Advanced Persistent Threat Solution   was not among those recently tested by NSS Labs in their April Breach Detection Study.   

Check Point was cited by Gartner as being the market share leader in firewall installed base. They offer an extensive line of security appliances and were also delivered the industry’s first flexible, extensible security architecture, the Check Point Software Blade Architecture.   Check Point’s Anti-Bot Software Blade detects bot-infected machines, prevents bot damages by blocking bot C&C communications. This isn’t a comprehensive Advanced Persistent Threat Solution, but it helps protect the network.

PAN’s product portfolio isn’t quite as extensive Check Point’s,   they do offer a virtualized firewall platform in addition to the more traditional appliance offering, threat subscriptions for URL filtering, and a management platform.  

Fortinet was rated a Challenger by Gartner. They stated Fortinet was “not often beating Leaders in mainstream enterprise selections based on features and vision, nor causing Leaders to react to Fortinet.”

Cisco was rated a Challenger as well.  Gartner didn’t seem them displacing   PAN nor Check Point on the basis of visions or features.  They saw Cisco winning firewall business through channel “execution and “aggressive discounting”.

Juniper Networks completed the trio of companies in the Challenger quadrant.  McAfee was a leader in the Niche quadrant.

Offerings from F5, Arkoon-Netasq, and AhnLab were the furthest down and to the left in the Magic Quadrant.

Check out the complete report.  For an assessment of all sixteen vendors in the report. Some names you’re familiar with may be missing due to consolidation. Gartner also has some brief information on why virtualized firewall penetration is a less than two percent.  “Security-minded enterprises are also rightly skeptical of running firewalls within a hypervisor that is between the threat and the firewall,” according to Gartner. 

 Regarding the Leaders quadrant from the Gartner Magic Quadrant Endpoint report - “A leading vendor isn't a default choice for every buyer, and clients should not assume that they must buy only from vendors in the Leaders quadrant.  Some clients believe that Leaders are spreading their efforts too thinly and aren't pursuing clients' special needs.”

For more details on the Magic Quadrant and how it is created, read “Magic Quadrants and MarketScopes: How Gartner Evaluates Vendors within a Market”.     Sometimes a leader is not the best solution for a particular customer.  Despite that, you will see many   presentations where the vendor uses being in the Leaders quadrant   as a reason to buy from that particular vendor.  Who would have thought that they would do that? www.gartner.com

Some of NSS Labs reports are available at no charge.  www.nsslabs.com

craig kensek

Fire in FireEye Valuation Gets Doused (slightly) With Release of NSS Breach Study Report – He Said, She Said Begins

 NSS Labs issued their Breach Detection Security Value Map on April 2  Neither FireEye nor AhnLab can be pleased.  In brief, the Value Map  measures security effectiveness on the Y-axis and Total Cost of Ownership (TCO) per protected MbPS on the X-axis.  AhnLab and FireEye finished in the dreaded lower left hand corner with FireEye coming in last in security effectiveness (AhnLab was close).  AhnLab had the highest TCO per Protected MBPS. The other four company’s products were in the upper right hand quadrant (Quadrant 1), Fidelis, Fortinet, Trend Micro, and SourceFire. They were all around 98% to 99% effective in NSS testing.  SourceFire was the winner, overall. 

From NSS, “Quadrant 1 contains those products that are recommended for both security effectiveness/management and value.  These devices provide a very high level of protection, manageability, and value for money.”  This document is publicly available from Fortinet as is a detailed report for their FortiSandbox 200D appliance.

http://www.fortinet.com/resource_center/whitepapers/breach-detection-systems-beyond-hype.html

Key findings mentioned in the press release - “Four of Six Leading Vendors Receive Coveted NSS ‘Recommended Rating’”

• Four of six products tested achieved over 95% in overall security effectiveness:   five of the six also received a 0% false positive rate.  AhnLab was the sixth with a 7% false positive rate.  FireEye had the lowest security effectiveness, around 94.5%. 
• Money Doesn’t Always Buy the Best Security: Total Cost of Ownership per Protected-Mbps ranged from $231 to $468 with the highest priced solution,   Conversely, Sourcefire (Cisco) had the lowest TCO and also received one of the highest security effectiveness ratings.
• All BDS Solutions Performed At or Above Vendor Throughput Claims

https://www.nsslabs.com/news/press-releases/nss-labs-reveals-first-security-value-map%E2%84%A2-breach-detection-systems

NSS Labs did not receive any compensation in return for vendor participation; All testing and research was conducted free of charge.

FireEye Stock Price (FEYE)

FireEye stock has dropped 49% percent from its March high of $97.35 to closing at $47.33 on April 11.  52-week range - $33.30 - $97.35.  It will be interesting now to see how the stock performs.  Q1 results won’t be announced until May 6.  Note -  The stock was at   $61.49 on April 2 when the report was released.  FireEye's  Q1 results won’t be comparable to    last year’s Q1 since revenue from their Mandiant acquisition after January 1 will be included.  The stock is up about 15% since the beginning of the year.  NASDAQ is down about 3% ovr the same period of time.

When you’re the market share leader, finishing low in an impartial test, one defense is to attack the attacker.

  

He Said - FireEye

"We are a vendor that specializes in advanced attack detection, not in detecting known, stale samples,” Gupta, FireEye Vice President of Products said.  "We ran their malware samples in our lab and detected every single one of them." A valid test would have used a zero-day exploit to evaluate the detection capabilities of the appliances or, at a minimum, the testing could have been done in a live, customer environment, Gupta added.

FireEye was quick to reply in a blog “Real World vs. Lab Testing: The FireEye Response to NSS Labs Breach Detection Systems Report” At a high level: 

• Issue #1:  Poor sample selection
• Issue #2:  Differing definitions of advanced malware
• Issue #3:  Poor test methodology.   

FireEye offered several paragraphs of detail for each of the above.  It is worth reading the blog.

http://www.fireeye.com/blog/corporate/2014/04/real-world-vs-lab-testing-the-fireeye-response-to-nss-labs-breach-detection-systems-report.html

“The best way to evaluate FireEye is for an organization to deploy our technology in their own environment and they will understand why we are the market leader in stopping advanced attacks, “said Dave Merkel, CTO in an April 2 Network World article.

She Said – NSS Labs

NSS Labs was also quick to replay in a blog “Don't Shoot the Messenger”

Their response is also good reading as most of the response consists of   a 20-bullet point “FireEye Claim” and “NSS Response” table.

“Not everyone can end up in the top right quadrant of the NSS Labs Security Value Map™ (SVM), so it is not unusual for someone to be unhappy.  It is, however, unusual for someone to behave the way FireEye did in this instance.  Normally we would not respond to such attacks, but there are a number of untruths and misdirection’s in their blog post that we feel we must address”, stated Bob Walder, President, and Chief Research Officer at NSS.  “FireEye’s results were not that bad.  The real issue here is that FireEye now has credible competition in the BDS market place and the data from this NSS test shows it.”

https://www.nsslabs.com/blog/dont-shoot-messenger

How Did This Begin

Three companies were tested last summer by NSS Labs in their initial breach study, AhnLab, FireEye, and Fidelis.  Fidelis made their report publicly available and challenged FireEye to do the same.  AhnLab issued a press release about their results, and in a blog went, “FireEye, hello?”  No press release by FireEye on their results.  Demerits to publications not asking about this!  With respect to the three companies, NSS has a multi-page document letting the firms tested know what they can do with the test results.  One thing they can’t do is start-doing comparisons with other companies, combining charts, et cetera from the reports.  The reports were available for purchase.

And What about NSS Labs’ Reputation?

In “IT Security Survey 2014” by  test group AV-Comparatives (www.av-comparatves.org),   issued in February, NSS Labs came in ninth out of 15 vendors.  Over 5800 users responded to the survey.  

 http://www.av-comparatives.org/security-usage-surveys/   

Timing Means Everything When Stock is Sold

On March 12, insider transactions of FireEye stock at $79.54 included: 

1. Norwest Venture Partners IX, LP sold 2 million shares, grossing $160 million.
2. FireEye CTO Aziz Ashar sold 1.04 million shares, grossing $83 million
3. FireEye CEO Dave DeWalt sold 486 thousand shares grossing $38 million

Insiders can’t sell shares whenever they want.  There are windows near the release of financial results that they can’t do anything.  A more comprehensive list of insider transactions can be viewed at

 http://finance.yahoo.com/q/it?s=FEYE+Insider+Transactions

  

It’s difficult to test security products.  Every environment is unique.  The best way for companies to evaluate products is to bring them in and to look at tests by reliable test groups.  The report by NSS Labs probably means   that FireEye will face more testing in house by potential vendors  rather than just be evaluated separately. 

Twitter - ckensek

February Virus Bulletin RAP Averages Quadrant (Reactive and Proactive) July 2012 through February 2013

Virus Bulletin has released their   RAP Averages Quadrant for the July through February timeframe.  There was a little bit of movement from the previous test, but nothing too exciting.  Avira Free wins among the companies best known for their freemium solutions.  Avast was a little ahead of AVG Technologies for Praha bragging rights.   

Once again, some estimating was necessary to pick the below.

RAP Averages Quadrant July 2012 through February 2013

1. Coranti
2. Avira Free
3. G Data
4. Fortinet
5. Lavasoft, TrustPort, BitDefender, BullGuard, Huari

Hall of Shame awards for their performance – Total Defense (by far), SPAMfighter, Frisk, and Commtouch. All scored below 70% on Reactive Detection, with Total Defense at about 55%.  Where are the 800-pound gorillas awards go to Symantec and Trend Micro for not being in the test.  Come on, Steve Bennett.  This test and AV-comparatives.  Time to step up.  Symantec had a big booth at the RSA security show.  The company   should be willing to be in these. 

The previous RAP averages test  had clusters of companies, as listed below. 

1. Zeobit, Coranti (clear winners)
2. Lavasoft, TrustPort, G Data
3. Fortinet, Avira Free, Avira Pro, Roboscan, BitDefender, BullGuard, Emisoft, eScan

 Tests like these provide  useful information in evaluating the relative strengths of the products.  It obviously wins out over the wisdom of Facebook fans clicking on like!  You can view the RAP Averages Quadrant chart at

http://www.virusbtn.com/vb100/latest_comparative/index

Subscribers to Virus Bulletin's publications have access to more details on the results.

RAP Averages Quadrant

This test measures products' detection rates across four distinct sets of malware samples.  The first three test sets comprise malware first seen in each of the three weeks prior to product submission.  These measure how quickly product developers and labs react to the steady flood of new malware emerging every day across the world.  A fourth test set consists of malware samples first seen in the week after product submission.

http://www.virusbtn.com/vb100/latest_comparative/index

  

Virus Bulletin

UK based Virus Bulletin started in 1989.  They provide PC users with a regular source of intelligence about computer viruses, their prevention, detection, and removal, and how to recover programs and data following an attack.  The Virus Bulletin website is at www.virusbtn.com

 

Palo Alto Networks Tosses the Gauntlet at Check Point Software Technologies

Palo Alto Networks is offering a $2,000 PA-2000 Next Generation Firewall (NGFW) appliance to qualified companies who take a meeting with them to discuss their solutions.  They also have a series of Five TechBuster videos comparing their NGFW’s to Check Point's products. These videos include Episode 2,  “Check Point Firewalls Have Better Price/Performance than Palo Alto Networks", and Episode Five,  “Check Point Application Control is as Easy to use as Palo Alto Networks”.  You have to love it when the 800-pound gorillas go mano a mano.  You also wonder what SonicWall is saying on the sidelines about all this. 

http://www.paloaltonetworks.com/cam/techbusters/pa-200/

According to Palo Alto Networks, the Palo Alto Networks™ PA-200 is targeted at high-speed firewall deployments within distributed enterprise branch offices.  The PA-200 manages network traffic flows using dedicated computing resources for networking, security, threat prevention, and management.

Palo Alto Networks outperformed  Check Point Software Technologies on the NSS Labs 2012 Next Generation Firewall Value Map.  This report was released during RSA 2012, San Francisco.  It is available online.  The report measures Block Rate versus Price per Protected-Mbps.  SonicWall also outperformed Check Point. 

 http://kensek.blogspot.com/2012/03/sonicwall-palo-alto-networks-top.html

 

What’s a little Next Generation name calling between friends? Particularly when the Palo Alto Networks founders came from Check Point.

Palo Alto Networks reported their fiscal Q1 2013 revenues during the first week of December.  Total revenue for the fiscal first quarter grew 50 percent year-over-year to $85.9 million, compared with $57.1 million in the fiscal first quarter of 2012.  They suffered a GAAP net loss for the fiscal first quarter of $3.5 million.  The market wasn’t pleased.  The stock fell below $47 shortly after the announcement after peaking around $72 in early December. 

 

It'll be a battle in  2013 in the NGFW marketplace.   SonicWall, owned by Dell,  has NGFW products that extend to the enterprise.  Fortinet has been claiming since January that they have the world’s fastest firewalls.  While Fortinet had a high Block Rate in the NSS test, their Price per Protect Mbps was the highest of any company’s product tested, with the exception of Juniper Networks.

October 2012 Virus Bulletin VB100 Awards and RAP Averages Quadrant (Reactive and Proactive)

Virus Bulletin has released their October 2012 VB100 testing and published their RAP Averages Quadrant for the April through October timeframe.  Neither test results were too exciting.  No companies going irate and withdrawing from further testing, in all likelihood. 

The VB100 testing was done on Windows Server 2003.  Only 30 products were in the test.  One third of the products failed.  There were no “major” vendors failing.  Emisoft has failed three out of the last four VB100 tests they have been in.  This can’t be fun for their marketing department.

There were three clusters of vendors who scored over 95% in Reactive Detection and 80% in proactive detection.  It’s not even worth eyeballing to rank within the clusters.  These vendors are (from first to third)

1. Zeobit, Coranti (clear winners)
2. Lavasoft, TrustPort, G Data 
3. Fortinet, Avira Free, Avira Pro, Roboscan, BitDefender, BullGuard, Emisoft, eScan

There was definitely some movement from the previous test. Congratulations,  in particular, to those in “1” and “2” above!

The Top 10 in the February through August test

1. Coranti (a clear first among the top 10)
2. Huari (a clear second among the top 10)
3. Tencent
4. Lavasoft
5. BitDefender
6. G Data
7. Avira Pro
8. TrustPort
9. Emisoft
10. Avira Free

Hall of Shame awards in the latest test for scoring below 70% on Reactive Detection and below 65% of Proactive Detection (from best to worst), Commtouch, Frisk, Iolo, Total Defense Business, and UnThreat.  No fun for these companies http://www.virusbtn.com/vb100/latest_comparative/index

Tests like these provide  useful information in evaluating the relative strengths of the products.  It obviously wins out over the wisdom of Facebook fans clicking on like!  You can view the  RAP Averages Quadrant chart at

http://www.virusbtn.com/vb100/latest_comparative/index

Subscribers to Virus Bulletin's publications have access to more details on the results.

RAP Averages Quadrant

This test measures products' detection rates across four distinct sets of malware samples.  The first three test sets comprise malware first seen in each of the three weeks prior to product submission.  These measure how quickly product developers and labs react to the steady flood of new malware emerging every day across the world.  A fourth test set consists of malware samples first seen in the week after product submission.

VB100 Test Methodology

The purpose of the VB100 comparative is to provide insight into the relative performance of the solutions taking part in the tests, covering as wide a range of areas as possible within the limitations of time and available resources.  More details are available at

http://www.virusbtn.com/vb100/about/methodology.xml

UK based Virus Bulletin started in 1989.  They provide PC users with a regular source of intelligence about computer viruses, their prevention, detection, and removal, and how to recover programs and data following an attack.  The Virus Bulletin website is at www.virusbtn.com