AMTSO Releases Additional Documents on Malware Security Test Design and Testing

The Anti-Malware Testing Standards Organization (www.amtso.org) recently released a pair of documents to assist the anti-malware testing and review community in test design and testing. Members of the AMTSO include both malware/internet security test groups as well as internet security providers such as Symantec, Panda, McAfee, and los free amigos, Avast, Avira, and AVG Technologies. The AMTSO currently has approximately 40 members.

These documents provide great guidelines (in a relatively brief format), for test organizations, publications doing testing, and individuals doing malware/internet security testing and reviews. Individuals and companies who use product reviews and round-ups to make a purchase decision can also benefit from these.

The “Performance Testing Guidelines” document is designed is to provide an overview of the issues involved in the accurate testing security technologies in terms of speed and resource usage.

Some of the closes to twenty factors for measurement discussed in the document include:

• File access time
• Memory usage
• CPU usage
• Network overhead

AMTSO stresses the need to run tests multiple times. Benchmarking a factor just once is inadequate. An average for multiple runs will minimizes the impact of anomalies and provides more accurate results.

The “Whole Product Testing” document discusses factors involved in designing and performing a complete security product test. This is versus isolating components of the product and performing a “Sum of the Parts” testing. They favor whole products tests, pointing out that product capabilities often work together to stop a given threat. This interaction cannot be shown through sum of the parts testing.

Factors they write about that need to be considered in designing and performing a test include:

• Stating the test purpose
• Selecting Samples
• Setting up Tests and Products
• Introducing Samples
• Handling User Interaction
• Capturing Test results
• Interpreting Test Results

While AMTSO designed these documents to assist the test community, the overall beneficiaries are home and business security users.