Methods for Personalized Design of Data Sonification

Abstract

This thesis examines the process of designing personalized sonifications -- audio representations of data -- from the perspective of the everyday end user, who is the ultimate listener and interpreter of that data. Designing sonifications is a complex task that involves domain knowledge of the data, principles of audio and sonification design, and the technical aspects of connecting the data to the sound. Typically, this knowledge is distributed between sound designers, programmers, and the end users, and the resulting sonifications require time to be developed and tested thoroughly. However, in many cases once the sonification has been created and deployed, the end user no longer has the opportunity to make changes to the design. Furthermore, if the design cannot be changed, it can be challenging for different end users to use, as each of them has individual preferences as to what is aesthetically pleasing, and individual differences in how audio representations of data are perceived and interpreted.

In this thesis, we address these challenges via several approaches. We leverage natural soundscapes from the environment (sounds of the weather, animal vocalizations, etc.) that have already been shown to have a pleasant aesthetic for conveying information and can be easily recognized and described by everyday people. We investigate the techniques for algorithmically generating and evaluating sonification designs. We develop user interfaces to allow end users to explore the sonification design space and assume ownership of their own design process.

The primary contributions of this work include: (1) novel models for representing data with environmental soundscapes that incorporate user interaction to make sonification design accessible to everyday people; (2) new algorithmic methods that use optimization and crowdsourcing to determine how to represent data with soundscapes sounds; (3) a demonstration of the benefits and challenges of using real world soundscape sounds for representing Twitter data; (4) a new performance paradigm that incorporates the audience as a part of the sound design process in musical performance; (5) a demonstration of the importance of control and interaction in end-user sonification design; and (6) a better understanding of the requirements and challenges in creating tools for end-user sonification design. This work enables everyday people to create their own, personal representations of data, and contributes to the broader perspective on sonification design.