# Don't Pick the Cherry: An Evaluation Methodology for Android Malware Detection Methods

**Authors:** Aleieldin Salem, Sebastian Banescu, Alexander Pretschner  
**arXiv:** [1903.10560](https://arxiv.org/abs/1903.10560)  
**Date:** 2019-03-25  
**Summary page:** https://www.summarizepaper.com/en/paper/1903.10560v1/

## Key points

- The malware research community relies on scan results from online platforms like VirusTotal for evaluating detection methods
- Lack of standards in interpreting data and labeling apps leads to variability in labeling schemes
- Variability in labeling and dynamic nature of VirusTotal's results impact accuracy of detection methods
- Authors address these challenges by demonstrating the effect of time, labeling schemes, and attack scenarios on an ensemble of Android repackaged malware detection methods called dejavu
- Evaluation conducted using over 30,000 real-world Android apps
- Results show how varying dimensions impact dejavu's performance
- Advocates for adoption of standardized methodology considering time, labeling schemes, and attack scenarios when evaluating new malware detection methods
- Importance of standardization emphasized for fair comparisons between techniques and improved accuracy in detecting Android malware.

## Summary

The malware research community heavily relies on scan results from online platforms like VirusTotal to evaluate detection methods. However, the lack of standards in interpreting this data and labeling apps leads researchers to rely on their own intuitions and adopt different labeling schemes. This variability in labeling, combined with the dynamic nature of VirusTotal's results, significantly impacts the accuracy of detection methods even when applied to the same dataset. As a result, it becomes challenging to objectively compare different malware detection techniques. In this paper titled "Don't Pick the Cherry: An Evaluation Methodology for Android Malware Detection Methods," authors Aleieldin Salem, Sebastian Banescu, and Alexander Pretschner address these challenges by demonstrating the effect of three dimensions on the performance of an ensemble of Android repackaged malware detection methods called dejavu. The three dimensions considered are time, labeling schemes, and attack scenarios. The authors conduct their evaluation using over 30,000 real-world Android apps. The results obtained clearly illustrate how varying these dimensions impacts dejavu's performance. By highlighting these effects, the authors advocate for the adoption of a standardized methodology that takes into account these three dimensions when evaluating newly-developed methods for detecting Android (repackaged) malware. Overall, this paper sheds light on the importance of considering time, labeling schemes, and attack scenarios in evaluating malware detection methods. It emphasizes the need for standardization in order to facilitate fair comparisons between different techniques and improve overall accuracy in detecting Android malware.