Biblio

Side-channel attacks occur on smartphone keystrokes, where the input can be intercepted by a tapping sound. Ilia et al. reported that keystrokes can be predicted with 61% accuracy from tapping sounds listened to by the built-in microphone of a legitimate user's device. Li et al. reported that by emitting sonar sounds from an attacker smartphone's built-in speaker and analyzing the reflected waves from a legitimate user's finger at the time of tap input, keystrokes can be estimated with 90% accuracy. However, the method proposed by Ilia et al. requires prior penetration of the target smartphone and the attack scenario lacks plausibility; if the attacker's smartphone can be penetrated, the keylogger can directly acquire the keystrokes of a legitimate user. In addition, the method proposed by Li et al. is a side-channel attack in which the attacker actively interferes with the terminals of legitimate users and can be described as an active attack scenario. Herein, we analyze the extent to which a user's keystrokes are leaked to the attacker in a passive attack scenario, where the attacker wiretaps the sounds of the legitimate user's keystrokes using an external microphone. First, we limited the keystrokes to the personal identification number input. Subsequently, mel-frequency cepstrum coefficients of tapping sound data were represented as image data. Consequently, we found that the input is discriminated with high accuracy using a convolutional neural network to estimate the key input.

Indoor localization has been a popular research subject in recent years. Usually, object localization using sound involves devices on the objects, acquiring data from stationary sound sources, or by localizing the objects with external sensors when the object generates sounds. Indoor localization systems using microphones have traditionally also used systems with several microphones, setting the limitations on cost efficiency and required space for the systems. In this paper, the goal is to investigate whether it is possible for a stationary system to localize a silent object in a room, with only one microphone and ambient noise as information carrier. A subtraction method has been combined with a fingerprint technique, to define and distinguish the noise absorption characteristic of the silent object in the frequency domain for different object positions. The absorption characteristics of several positions of the object is taken as comparison references, serving as fingerprints of known positions for an object. With the experiment result, the tentative idea has been verified as feasible, and noise signal based lateral localization of silent objects can be achieved.

Voice user interfaces can offer intuitive interaction with our devices, but the usability and audio quality could be further improved if multiple devices could collaborate to provide a distributed voice user interface. To ensure that users' voices are not shared with unauthorized devices, it is however necessary to design an access management system that adapts to the users' needs. Prior work has demonstrated that a combination of audio fingerprinting and fuzzy cryptography yields a robust pairing of devices without sharing the information that they record. However, the robustness of these systems is partially based on the extensive duration of the recordings that are required to obtain the fingerprint. This paper analyzes methods for robust generation of acoustic fingerprints in short periods of time to enable the responsive pairing of devices according to changes in the acoustic scenery and can be integrated into other typical speech processing tools.

This paper addresses the fundamental problem of identifying driver talking directions using a single smartphone, which can help drivers by warning distraction of having conversations with passengers in a vehicle and enable safety enhancement. The basic idea of our system is to perform talking status and direction identification using two microphones on a smartphone. We first use the sound recorded by the two microphones to identify whether the driver is talking or not. If yes, we then extract the so-called channel fingerprint from the speech signal and classify it into one of three typical driver talking directions, namely, front, right and back, using a trained model obtained in advance. The key novelty of our scheme is the proposition of channel fingerprint which leverages the heavy multipath effects in the harsh in-vehicle environment and cancels the variability of human voice, both of which combine to invalidate traditional TDoA, DoA and fingerprint based sound source localization approaches. We conducted extensive experiments using two kinds of phones and two vehicles for four phone placements in three representative scenarios, and collected 23 hours voice data from 20 participants. The results show that our system can achieve 95.0% classification accuracy on average.

With the increasing prevalence of mobile devices, face-to-face device-to-device (D2D) communication has been applied to a variety of daily scenarios such as mobile payment and short distance file transfer. In D2D communications, a critical security problem is verifying the legitimacy of devices when they share no secrets in advance. Previous research addressed the problem with device authentication and pairing schemes based on user intervention or exploiting physical properties of the radio or acoustic channels. However, a remaining challenge is to secure face-to-face D2D communication even in the middle of a crowd, within which an attacker may hide. In this paper, we present Nhuth, a nonlinearity-enhanced, location-sensitive authentication mechanism for such communication. Especially, we target at the secure authentication within a limited range such as 20 cm, which is the common case for face-to-face scenarios. Nhuth contains averification scheme based on the nonlinear distortion of speaker-microphone systems and a location-based-validation model. The verification scheme guarantees device authentication consistency by extracting acoustic nonlinearity patterns (ANP) while the validation model ensures device legitimacy by measuring the time difference of arrival (TDOA) at two microphones. We analyze the security of Nhuth theoretically and evaluate its performance experimentally. Results show that Nhuth can verify the device legitimacy in the presence of nearby attackers.

To experimentally study the characteristics of ultrasonic sensors, a wideband air-coupled ultrasonic transducer, wideband electromagnetic dynamic acoustic transducer (WEMDAT), is designed and fabricated. Characterisation methods, including electrical impedance analysis, laser Doppler vibrometry and pressure-field microphone measurement, are used to examine the performance of the WEMDAT, which have shown that the transducer has a wide bandwidth ranging approximately from 47 kHz to 145 kHz and a good directivity with a beam angle of around 20˚ with no evident side lobes. A 40 kHz commercial flexural ultrasonic transducer (FUT) is then taken as an example to receive ultrasonic waves in a pitch-catch configuration to evaluate the performance of the WEMDAT as an acoustic source. Experiment results have demonstrated that the WEMDAT can maintain the most of the frequency content of a 5 cycle 40 kHz tone burst electric signal and convert it into an ultrasonic wave for studying the dynamic characteristic and the directivity pattern of the ultrasonic receiver. A comparison of the dynamic characteristics between the transmitting and the receiving processes of the same FUT reveals that the FUT has a wider bandwidth when operating as an ultrasonic receiver than operating as a transmitter, which indicates that it is necessary to quantitatively investigate the receiving process of an ultrasonic transducer, demonstrating a huge potential of the WEMDAT serving as a standard acoustic source for ultrasonic sensors for various air-coupled ultrasonic applications.

Human computer operations such as writing documents and playing games have become popular in our daily lives. These activities (especially if identified in a non-intrusive manner) can be used to facilitate context-aware services. In this paper, we propose to recognize human computer operations through keystroke sensing with a smartphone. Specifically, we first utilize the microphone embedded in a smartphone to sense the input audio from a computer keyboard. We then identify keystrokes using fingerprint identification techniques. The determined keystrokes are then corrected with a word recognition procedure, which utilizes the relations of adjacent letters in a word. Finally, by fusing both semantic and acoustic features, a classification model is constructed to recognize four typical human computer operations: 1) chatting; 2) coding; 3) writing documents; and 4) playing games. We recruited 15 volunteers to complete these operations, and evaluated the proposed approach from multiple aspects in realistic environments. Experimental results validated the effectiveness of our approach.

Indoor localization of unknown acoustic events with MEMS microphone arrays have a huge potential in applications like home assisted living and surveillance. This article presents an Angle of Arrival (AoA) fingerprinting method for use in Wireless Acoustic Sensor Networks (WASNs) with low-profile microphone arrays. In a first research phase, acoustic measurements are performed in an anechoic room to evaluate two computationally efficient time domain delay-based AoA algorithms: one based on dot product calculations and another based on dot products with a PHAse Transform (PHAT). The evaluation of the algorithms is conducted with two sound events: white noise and a female voice. The algorithms are able to calculate the AoA with Root Mean Square Errors (RMSEs) of 3.5° for white noise and 9.8° to 16° for female vocal sounds. In the second research phase, an AoA fingerprinting algorithm is developed for acoustic event localization. The proposed solution is experimentally verified in a room of 4.25 m by 9.20 m with 4 acoustic sensor nodes. Acoustic fingerprints of white noise, recorded along a predefined grid in the room, are used to localize white noise and vocal sounds. The localization errors are evaluated using one node at a time, resulting in mean localization errors between 0.65 m and 0.98 m for white noise and between 1.18 m and 1.52 m for vocal sounds.

We propose $μ$Leech, a new embedded trusted platform module for next generation power scavenging devices. Such power scavenging devices are already widely deployed. For instance, the Square point-of-sale reader uses the microphone/speaker interface of a smartphone for communications and as power supply. While such devices are used as trusted devices in security critical applications in the wild, they have not been properly evaluated yet. $μ$Leech can securely store keys and provide cryptographic services to any connected smart phone. Our design also facilitates physical security analysis by providing interfaces to facilitate acquisition of power traces and clock manipulation attacks. Thus $μ$Leech empowers security researchers to analyze leakage in next generation embedded and IoT devices and to evaluate countermeasures before deployment.