Interacting with Music in Video Games

Derek Dahmer <dahmerda (at) seas (dot) upenn (dot) edu>


A musical track has been a part of nearly all commercially sold video games created in the past decade. Yet as essential as it is, the musical track is almost always relegated to the background, only affected by macro changes in the game such as the shifting of levels or substantial plot changes. A new concept emerging in the independent games genre has been making the musical component a substantially more prominent aspect of the game experience. Games such as Rez and Everyday Shooter have attempted this by tying user actions to changes in the musical track, as well as by showing certain characteristics of the active sound track in the surrounding environment. The result has been a markedly more immersive, expressive and unique experience for the player that has manifested in both game sales and critical acclaim. This project seeks to further the development of the focus on audio in two ways.

First is by expanding on the idea of having a sound track change as the result of specific user actions. In particular, the visual-audio connection of game events can be further strengthened, creating a distinct relationship for the player between potential actions and potential musical outcomes. A distinct relationship between action and audio will serve to increase overall immersion in the game since audio manipulation has the potential to feel like more of a natural extension of gameplay than numbers on a screen or a prerecorded sound clip. In addition, the player will feel a stronger connection to game as it becomes apparent that each action choice has a controllable outcome on the previously uncontrollable sound track.

Second, this project will explore using audio as a means of giving the player additional information that will help or even be essential in non-musical gameplay decisions. In particular, states of the active entities in the game, such as players and enemies, can be represented through audio.

The goal is to create a need for the player to be conscious of the state of the audio in order to succeed in the game, the same way visual awareness is necessary in nearly every existing game today.