The psychology of music in multimedia

For most of the history of film-making, music has played an integral role serving many functions - such as conveying emotion, heightening tension, and influencing interpretation and inferences about events and characters. More recently, with the enormous growth of the gaming industry and the Internet, a new role for music has emerged. However, all of these applications of music depend on complex mental processes which are being identified through research on human participants in multimedia contexts. The Psychology of Music in Multimedia is the first book dedicated to this fascinating topic. The Psychology of Music in Multimedia presents a wide range of scientific research on the psychological processes involved in the integration of sound and image when engaging with film, television, video, interactive games, and computer interfaces. Collectively, the rich chapters in this edited volume represent a comprehensive treatment of the existing research on the multimedia experience, with the aim of disseminating the current knowledge base and inspiring future scholarship. The focus on empirical research and the strong psychological framework make this book an exceptional and distinctive contribution to the field. The international collection of contributors represents eight countries and a broad range of disciplines including psychology, musicology, neuroscience, media studies, film, and communications. Each chapter includes a comprehensive review of the topic and, where appropriate, identifies models that can be empirically tested. Part One presents contrasting theoretical approaches from cognitive psychology, philosophy, semiotics, communication, musicology, and neuroscience. Part Two reviews research on the structural aspects of music and multimedia, while Part Three focuses on research examining the influence of music on perceived meaning in the multimedia experience. Part Four explores empirical findings in a variety of real-world applications of music in multimedia including entertainment and educational media for children, video and computer games, television and online advertising, and auditory displays of information. Finally, the closing chapter in Part Five identifies emerging themes and points to the value of broadening the scope of research to encompass multisensory, multidisciplinary, and cross-cultural perspectives to advance our understanding of the role of music in multimedia. This is a valuable book for those in the fields of music psychology and musicology, as well as film and media studies.

Siu-Lan Tan is Associate Professor of Psychology at Kalamazoo College in Michigan USA. Born in Indonesia and raised in Hong Kong, she holds diplomas from the Associated Board of the Royal Schools of Music (England) and degrees in Music and Piano Pedagogy, and taught music in Hong Kong and California for many years before completing a PhD in Psychology at Georgetown University USA and a term on scholarship at Oxford University in England. Tan's research on video games and virtual reality games has been published in the Journal of Applied Developmental Psychology, International Journal of Gaming and Computer-Mediated Simulations, Interacting with Video (Praeger 1996), and Interdisciplinary Advancements in Gaming, Simulations and Virtual Environments (IGI 2012). Her research on film music and other topics has appeared in Music Perception, Psychology of Music, Psychomusicology: Music, Mind, & Brain, and College Music Symposium.
Annabel J. Cohen is Professor of Psychology at the University of Prince Edward Island in Canada. Her doctorate is from Queen's University, B.A from McGill, and Associate diploma (ARCT) from the Royal Conservatory of Music, Toronto. She has dedicated her career to the study of music perception and cognition, with extensions to multimedia. She has published on the topics as tonality, music transposition, the acquisition of music grammar, and film music, from a cognitive perspective. She has led multi-institutional research projects focusing on harnessing multimedia for education in the context of culture and cognition. On that foundation, she currently directs a major international collaborative research initiative (AIRS - Advancing Interdisciplinary Research in Singing). Cohen is the Editor of Psychomusicology: Music, Mind, & Brain and serves on the consulting boards of several journals (e.g., Music and the Moving Image, Musicae Scientiae, Music Perception, Psychology of Music, The Soundtrack).
Scott D. Lipscomb is Associate Professor and Division Head of Music Education and Music Therapy and Associate Director for the School of Music at the University of Minnesota - Twin Cities, USA. In addition to his primary research interest in "sound for multimedia," he is currently pursuing investigations related to surround sound presentation of cinema and musical sound, the effect of music in video game contexts, the impact of visual performance information on listener preference for "complex" music, integration of music across the K-12 curriculum, and the development of interactive instructional media to enhance the music learning experience. Lipscomb served two three-year terms as Treasurer for the Society for Music Perception and Cognition, four two-year terms as President of the Association for Technology in Music Instruction (ATMI), and serves as a member of the Executive Board and as Chair of the Research Committee for TI:ME (Technology Institute for Music Educators).
Roger A. Kendall is a Professor in the Department of Ethnomusicology at the University of California, Los Angeles, USA in the specialization of Systematic Musicology. He co-authored an invited chapter on music perception and cognition for the Ecological Psychology volume of the Handbook on Perception, as well as a chapter on comparative music psychology for Psychology of Music (D. Deutsch, Ed. 2nd edition). His current research interests include comparative perceptual and acoustical analyses of natural versus synthetic and sampled orchestral timbres and spectra, tuning models and perception of the slendro mode in the Gamelan, expressive music performance modelled in terms of communication theory, and perception of meaning in film music. Kendall co-edited Perspectives in Systematic Musicology in Selected Reports in Ethnomusicology, Volume 12 and contributed a chapter outlining connections of experimental empirical research in meaning to visual and musical elements.

- Foreword - 1: Annabel J. Cohen, Scott D. Lipscomb, Siu-Lan Tan, and Roger A. Kendall: Introduction: The psychology of music in multimedia - Part I. Models and Multidisciplinary Perspectives - 2: Annabel J. Cohen: Congruence-Association Model of music and multimedia: Origin and evolution - 3: Roger A. Kendall and Scott D. Lipscomb: Experimental semiotics applied to visual, sound, and musical structures - 4: Mark Shevy: Integrating media effects research and music psychology - 5: David Bashwiner: Musical analysis for multimedia: A view from music theory - 6: Lars Kuchinke, Hermann Kappelhoff, and Stefan Koelsch: Music and emotion in narrative films: A neuroscientific perspective - Part II. Cross-modal Relations in Multimedia - 7: Shin-ichiro Iwamiya: Perceived congruence between auditory and visual elements in multimedia - 8: Zohar Eitan: How pitch and loudness shape musical space and motion - 9: Scott D. Lipscomb: Cross-modal alignment of accent structures in multimedia - Part III. Interpretation and Meaning - 10: Marilyn G. Boltz: Music videos and visual influences on music perception and appreciation: Should you want your MTV? - 11: Berthold Hoeckner and Howard C. Nusbaum: Music and memory in film and other multimedia: The Casablanca effect - Part IV. Applications: Music and Sound in Multimedia - 12: Sandra L. Calvert: Children's media: The role of music and audio features - 13: Mark Grimshaw, Siu-Lan Tan, and Scott D. Lipscomb: Playing with sound: The role of music and sound effects in gaming - 14: Mark Shevy and Kineta Hung: Music in television advertising and other persuasive media - 15: Agnieszka Roginska: Auditory icons, earcons, and displays: Information and expression through sound - 16: Mark Kerins: Understanding the impact of surround sound in multimedia - Part V. Future Research Directions - 17: Siu-Lan Tan, Annabel J. Cohen, Scott D. Lipscomb, and Roger A. Kendall: Future research directions for music and sound in multimedia