Android’s fast-lived development cycles and increasing amounts of manufacturers and device models make a comparison of relevant security attributes, in addition to the already difficult comparison of features, more challenging. Most smartphone reviews only consider offered features in their analysis. Smartphone manufacturers include their own software on top of the Android Open Source Project (AOSP) to improve user experience, to add their own pre-installed apps or apps from third-party sponsors, and to distinguish themselves from their competitors. These changes affect the security of smartphones. It is insufficient to validate device security state only based on measured data from real devices for a complete assessment. Promised major version releases, security updates, security update schedules of devices, and correct claims on security and privacy of pre-installed software are some aspects, which need statistically significant amounts of data to evaluate. Lack of software and security updates is a common reason for shorter lifespans of electronics, especially for smartphones. Validating the claims of manufacturers and publishing the results creates incentives towards more sustainable maintenance and longevity of smartphones. We present a novel scalable data collection and evaluation framework, which includes multiple sources of data like dedicated device farms, crowdsourcing, and webscraping. Our solution improves the comparability of devices based on their security attributes by providing measurements from real devices.
