Project Objectives:

As the form factor of our mobile and wearable devices shrinks, there exists an increasing need to support interaction beyond the confines of the device itself. Particularly on wearable devices, small touchscreens and interfaces can render complex input cumbersome. This project pushes the limits of vibration-based sensing to determine the location of a touch on extended surface areas as well as identify the object touching the surface leveraging a single sensor. Unlike capacitive sensing, it does not require conductive materials and compared to audio sensing it is more robust to acoustic noise. It supports a broad array of applications as shown in Figure 1 through either passive or active sensing using only a single sensor. The vibration-based sensing technique can further enable user authentication via finger inputs on ubiquitous surfaces, as shown in Figure 2. It integrates passcode, behavioral and physiological characteristics, and surface dependency together to provide a low-cost, tangible and enhanced security solution.

Technology Rationale:

Physical vibration is a mechanical phenomenon, which creates a mechanical wave transferring the initial energy through a medium. Similar to the transmission of wireless signals, when a vibration signal travels through a medium, it experiences attenuation along the propagation path and refection/diffraction when the signal hits the boundary of two different media (e.g., the contacting area between an object/finger and a medium). When the vibration signal hits the contacting area of the object/finger, part of the signal reflects back to the surface and the rest of it propagates into the object/finger (i.e., absorption) and bounces back to the surface along a different propagation path. The vibration signal travels different paths before reaching the receiver (i.e., vibration sensor). Such an attenuation diversity reflects the details of the type of material and path the vibration signal goes through, resulting in rich sensing information. In addition, regarding to finger-input based authentication, we find that the frequency response of the same user finger-press presents higher correlation than that of different users when they touch the same location on a surface. This important observation suggests that the vibration propagation properties are strongly influenced by unique human physical traits such as contacting area, touching force and etc., which can assist ubiquitous user authentication together with passcode on any surface beyond touch screens.

Technical Approach:

Single-Sensor Vibration Sensing. We develop the system based on a low cost single sensor prototype we built to transmit/receive vibration signals. It can be attached to non-conductive surfaces such as a table or a door and sense touching objects or users. By relying on vibrating signals, the system is less susceptible to environmental interferences from acoustic or radio-frequency noise. Specifically, the system takes as input time-series amplitude measurements of vibration signals from a vibration receiver. After receiving the vibration signals, the system performs Vibration Detection & Segmentation to detect and obtain the useful segment of the received vibration signals. In the profiling phase, the extracted features are considered to be the unique signature corresponding to the characteristics of the object's touches on the medium, for example, keystrokes' locations, or the weight and size of a smartphone on a nightstand. These features are labeled with corresponding ground truth (i.e., location, object type, etc.), and saved to build an object profile. In the identification phase, the collected vibration samples are used to extract vibration features, which serve as inputs to a vibration classifier via Vibration Classification based on SVM. The classifier compares the extracted features with the signatures in the pre-constructed profile to identify the target object and determine its location.


Finger-input based Authentication over Any Surface through a Single Sensor. In this work, we introduce a new authentication system grounded on low-cost, low-power tangible user interface, which has the flexibility to be deployed on ubiquitous surfaces. The system can authenticate the legitimate user and reject attacks well because of the following insights: 1) our study shows that vibration signals have the capability to perform cm-level location discrimination; and 2) unique features are embedded in a user's finger pressing at different locations on a solid surface. Such unique features reflect the characteristics of the user's finger touching on the medium (e.g., a door panel or a desk surface) including locations of touching, contacting area, touching force, and etc., making them capable to discriminate different touching locations of the same user and different users when touching on the same location. Thus, the proposed system enables users to finger-input (i.e., touch or write) on solid surface and is robust to passcode theft or passcode cracking by integrating 1) passcode, 2) behavioral and physiological characteristics (e.g., touching force and contacting area), and 3) surface dependency (e.g., house door or office desk) together to provide enhanced security.

Results To Date and Future Work Plan:

This project has led to papers in IEEE SECON'17 (Best Paper Award) and ACM CCS'17. We have reported both system evaluation performance on the vibration-based sensing and user authentication. Figure 3 and Figure 4 show the example performance results of keystroke recognition accuracy (virtual keyboard application) and PIN-based user authentication, respectively. This project has been reported in Rutgers News: "Good Vibrations: Smart Access to Homes and Cars Using Fingers" and over 30 media outlets, e.g., IEEE Spectrum, Yahoo Finance News, NSF Science 360 News, and Futurity.