Automatic testing of smart speaker apps

Abstract

With the emergence of the Internet of Things (IoT), Smart Speakers open up a new world where we can talk to a machine for getting help in our day-to-day lives. The Smart Speaker Apps (SSA)s provide a user-friendly vocal experience to the customers by allowing them to dictate commands to the speaker through voice commands. Amazon Alexa is one of the most prevalent smart speakers which allows third-party developers to write SSAs called Skills. Due to the prevalence of Alexa, it has become vulnerable to security and privacy threats by malicious skill developers. In particular, Alexa skills could be overprivileged such that they collect more data than necessary or specified by the privacy policy in the skills description. In this research, we systematically explore skills to test whether the behaviors of the skills adhere to the privacy policy provided in the skill description. We extracted the utterances related to privacy-sensitive behavior of the skills through Natural Language Processing (NLP) techniques. Second, we implemented a dynamic testing tool Test case Generator & Invocator based on the fuzzing technique to automatically manipulate the inputs to the skills and observe the output to identify the skills which accept the privacy-sensitive information. During the study, we discovered that 21% of the tested skills accept privacy-sensitive data. We have simply focused on the real or actual behavior of the skills during the research. The claimed behavior of the skills is covered by our study, which will be the focus of further work.