Augmented Reality

I usually shy away from anything too technological but I'm thinking of incorporating an element of 'augmented reality' with this project now. Bit of a current buzzword [despite being coined as far back as 1990], it refers to the combination of real and virtual worlds.

What has particularly captured my imagination are the images or 'markers' that are used to trigger the virtual elements: some of them can have the appearance of codes or pixelated notation. So I'm wondering if I can create and use any visualizations from our 3Dmax-oriented class that bear a relation to the particular library or noise to use via these keys. Then the trigger itself can be incorporated within any print outputs to be used with a webcam, for example, to generate this other digital entity.

Decibel Dingbats by Sokratype

Available (for free) over at dafont.com.

There are also a couple of other dingbats based on sound/music notation featured on the same site.

Sounddesign by Mueller,Weiland

From the wonderful http://www.muellerweiland.de/


Sounddesign

Deutsches Filmmuseum
(Frankfurt am Main)

Ausstellungskonzept für 2011/2012

Das Projekt entstand in Kooperation zwischen dem Deutschen Filmmuseum, der Fachhochschule Düsseldorf und dem IMM der Robert-Schumann-Hochschule Düsseldorf. Im Austausch mit Sounddesignern wurde ein Corporate Sound, ein medienübergreifendes Corporate Design, eine Ausstellung, sowie ein begleitender Katalog konzipiert und gestaltet.

Zusammenarbeit mit Tobias Jochinke und Marc Rogmans www.sounddesign-exhibition.de

Studio Tonne

Some more remarkable, recommended work [with the tip coming from Tash] is that of Studio Tonne. Loads of great stuff on their web that touches on the relationship between sound and vision: from music packaging to interactive audio projects (or, as they prefer "sound toys").

Website here: http://www.studiotonne.com/

Bicycle Built for Two Thousand by Aaron Koblin and Daniel Massey

Bicycle Built for Two Thousand from Aaron on Vimeo.

The excellent link above came courtesy of Amy [cheers!] with the original project documented here: http://www.bicyclebuiltfortwothousand.com

Form+Format's video for the Shlohmo remix of Robot Koch

"GOROM SEN" SHLOHMO RMX from Form+Format on Vimeo.

Music sales by format data visualization

From the New York Times and demonstrating music sales as a truncated waveform for each particular format.

Enhanced screengrab of Toulon Library visualization video

Very much following on from the last post, here's a screengrab from one part of the video that has had some of its fuzziness cleaned up a little in Photoshop. There were some much nicer colour combinations going on within the video but this is simply to record a technique that can document some of the outputs for future reference.

One other point worth making is that it might be best to record those visualizations in the dark as there is a fair amount of glare coming from the screen.

Toulon Library recording with WMP visualization

A little experiment with the first of the Toulon Library recordings where it was played on the Windows Media Player with the 'Rhythmic Colours' visualization.

I've then recorded that playback as a video before attaching the original (better quality) recording as audio. Finally, the imagery has been filtered/pixelated so that it makes a more grid-like approximation of the sound visualization as coloured squares that actually grow in size as the video progresses. At times, the seemingly random colour combinations prove interesting.

It's a relatively quick exercise (even with the Youtube upload) but I'm wondering about its application as a more fully realised and perhaps sharper visual project that takes on new meaning without the audio track. (Obviously, an idea of how this might work is achieved by simply muting the volume.)

Toulon Library recordings as waveforms

I downloaded the recordings I made in libraries in both Toulon and Manchester as wav. files before opening them in Sonic Foundry ACID (a hardly state-of-the-art music package).

The attached images are some screen shots from the Toulon samples which provide a visualization of the encountered noise. But while I had the ACID software open, I started thinking about whether there's more that can be done with these sounds as a composition. The thing that puts me off that idea is that I want to do something that is cross-sensory [where a visual depiction describes an auditory form] and the whole 'found sound' thing has been done to death by people far more musically talented than I ever will be. Still, ACID - which I've used to edit sound for previous video projects - provides the opportunity to join, reconfigure and manipulate some of these recordings so there may still be the potential to do something new with these very raw samples.

Once I've done a little investigation into some of the different options in uploading playable sound files to this blog, I'll get the actual audio up here too.

Cymatics #3

Music notation: suggested teaching methods

At a more basic level, I came across the following resource for teachers which suggests different ways to familiarise pupils with music notation (http://www.name2.org.uk/proj/h2t6.php). Some of these methods could be adapted to suit other representations of sound/noise.

Hard-to-teach area 6:

Teaching notation

Teacher feedback: Some teachers say they find it hard to teach notation and motivate pupils. There was evidence that this is perhaps because their approach is based on their own very traditional background. There is a tendency toward over-reliance on standard notation without due consideration to whether it is the best for the learning context.

KS3 context: The English National Curriculum for Key Stage 3 music includes a requirement to study "staff notation and other relevant notations in a range of musical styles genres and traditions" (3d). In KS2 pupils will almost certainly have encountered graphic notation and graphics software, but many may not be familiar with standard notation at transition. In KS4 familiarity with standard notation makes it easier to achieve higher examination grades.

Index of teaching notation ideas

• Principle 1: Ensure a musical objective
• Principle 2: Consider an informal learning context
• Principle 3: Always keep notation appropriate to need
  
• Idea 1: Use waveforms as notation
• Idea 2: Use a piano grid / pitch contour as notation
• Idea 3: Use chord symbols
• Idea 4: Use world music notations
• Idea 5: Use graphic and pupil-devised notations
• Idea 6: Use standard notation software wisely
• Idea 7: Consider interactive notation activities
  
• Appendix: Other links submitted by teachers and publishers

Mention of specific products does not imply endorsement by Becta or NAME, who take no responsibility for the content and accuracy of external websites.

---

Principle 1: Ensure a musical objective

A KS3 lesson should not narrowly set out to 'teach music notation' for its own sake. This lacks true musical purpose and relevance. Instead, the learning task should have a real musical objective and, if encountered at all, standard notation should be of some genuine benefit in achieving that objective. If there is a more appropriate way of achieving the objective then standard notation probably has no place.

Back to index of teaching notation ideas

---

Principle 2: Consider an informal learning context

Notation is used in various ways throughout Musical Futures models, but is brought in after practical activity has been introduced. Pupils develop an understanding of why notation is useful – for example because they need to make a record of what they have been working on in a lesson. The teacher can then show them a relevant way of doing this.

In the informal learning model, where student are directing their own learning, they often request that they source notation, guitar tab, chords etc, usually from the Internet. They start to develop an understanding of notation in this way because it is meaningful to them. It is directly relevant to the activity they are pursuing and the goals they want to achieve. Find out more about the Musical Futures approach at www.musicalfutures.org.uk.

The Internet is a rich source of notations of many types, especially chords, lyrics and guitar tabs for well-known rock and pop songs. Just use any standard web search engine and search for song titles.

Back to index of teaching notation ideas

---

Principle 3: Always keep notation appropriate to need

Bear in mind that there are many ways of representing sound visually: waveforms, grids, contours, chord symbols, guitar tablature, sol-fa, etc. (some of these are described in the ideas below). Ask yourself whether you are using the most appropriate kind of notation for the actual learning need. An interesting case study comparing the effect of using graphic, staff and waveform notations for the same learning task appears in the KS Music Strategy materials at http://www3.hants.gov.uk/music/unit4-p17.htm

Back to index of teaching notation ideas

---

Idea 1: Use waveforms as notation

A waveform is a scientific representation of a sound, as shown below.

Waveforms require very little explanation to pupils. They represent much less of a hurdle to understanding than standard notation but are limited in the amount of detail they convey. Depending on the horizontal zoom, a typical waveform can indicate:

• when instruments start and stop playing (wave shape or flat line).
• whether a sound is quiet (shortish waves) or loud (taller waves).
• overall dynamic shapes and where a dynamic climax occurs.
• rhythms on a proportional linear scale (but time-related, not beat-related).

Examples:

1. Graphics derived from sound waves are used in many sound editing programs, e.g. Audacity, which is available for PC, Mac and Linux and can be downloaded free of charge from http://audacity.sourceforge.net/download.
2. In this Powerpoint presentation (12.3MB - this will take some time to load!) waveforms were chosen as the most accessible way to guide listening to the interplay of solo and ripieno in a concerto model. The resource is from a year 9 module called 'Concerto Conversations', which is part of the Practical Support Pack at www.teachernet.gov.uk/supportpack/module.aspx?t=2&s=12&y=42,43,44&p=&m=21562
3. Waveform display was chosen for the Sound Explorer user interface developed specifically for the SoundJunction website (standard notation is also available by clicking a button). Read about the Explorer tool at www.soundjunction.org/explorertoolhelp.aspa, then click the Sound Explorer link in the left column to use it yourself. Why do you think waveform notation was chosen for the main display?

To make your own waveform graphic:

1. Record or load your audio file into any program that displays it as a waveform.
2. Adjust the horizontal and vertical zoom to suit your purpose.
3. Use any screen capture program to save your chosen area of the screen as a graphics file.
4. That graphics file can now be imported into a word processor, presentation, publishing program, etc.

Back to index of teaching notation ideas

---

Idea 2: Use a piano grid / pitch contour as notation

Piano grids are provided in nearly all MIDI sequencers. They are fairly intuitive to understand, even to non-pianists. The horizontal scale (zoom) can be changed for different purposes but, unlike standard notation, symbol width is exactly proportional to duration. Piano grids convey precise information about:

• individual note pitches, including within chords (vertical position)
• beat position (horizontal position)
• note length and articulation (symbol width and separation)
• dynamics (not indicated below, but sometimes shown in a separate area under the grid or within each individual symbol).

The diagram below shows note lengths and articulation with more precision than is possible using standard notation (with none of the arcane squiggles and dots). Syncopation can also be shown without complications. Using a sequencer's piano grid edit screen is much more straightforward than its score edit screen. Try notating the phrase below as standard notation and consider how much more knowledge that requires, with no added musical insight.

Mixcraft's piano grid editor (www.acoustica.com/mixcraft)

Pitch contours often look like reduced piano grids. They are usually drawn without the grid and sometimes with non-horizontal lines and curves that indicate ongoing pitch directions rather than discrete events. They sacrifice precision for a less cluttered overall impression.

Matching pitch contour shown on Mixcraft's arrange screen

SoundJunction's Note Canvas tool (www.soundjunction.org/notecanvastool.aspa) is a free online tool intended for educational use. It shows a piano grid and also overlays a sloping contour shape. It is intended that this tool will eventually include a standard notation view option. You can discover some of its other features by opening the Note Canvas tool demo from the link in the introductory text on that page.

To make your own grid/contour graphic

• Adapt the equivalent advice offered in idea 1.

Back to index of teaching notation ideas

---

Idea 3: Use chord symbols

Chord symbols are most appropriate for activities when a harmonic framework is required but there is a need for some freedom of choice in how the chords are realised. Chord symbols are also helpful to indicate a context for more experienced melodic improvisers.

Band in a Box (www.band-in-a-box.com) is available for Windows and Mac computers. It is a powerful program that can:

• generate backing accompaniments to user-defined chord progressions in a broad range of musical styles.
• analyse audio recordings and generate matching chord symbols.
• play its own 'intelligent' melodic improvisations according to settings chosen by the user.
• record live input via a microphone or MIDI instrument.
• suggest chords and alternatives that would match a given melody.
• display its realisations using standard notation.
• do many other things! (browse the website for more information)

These features allow users to make creative musical decisions without needing a sophisticated existing knowledge of standard notation. This can keep motivation buoyant. Inspecting the resulting notation can then help to improve understanding in a way that is relevant to a musical task. Scores can also be printed out for use in live performance.

Band in a Box screen content can appear quite complex and KS3 pupils will need clear advice and guidance if they are to use it successfully on their own. This is one of the programs included in an optional face-to-face ICT component of the Trinity/OU CPD programme for KS2 music teachers. A time-limited free demo version is available for download from the Band in a Box website and there are several video tutorials there to get you started.

To make your own chord symbol sheet

• For simple chord symbols, any word processor will probably do fine.
• Alternatively, you can print out more advanced chord symbols using software such as Band in a Box or from many standard music notation programs (see idea 6 below).

Back to index of teaching notation ideas

---

Idea 4: Use world music notations

Bear in mind that the Key Stage 3 curriculum specifies 'notations' in the plural. Many localised styles and genres across the world have developed their own notation conventions. These are usually very efficiently matched to the features of the particular style. Unfortunately they are not generally amenable for ICT handling without a dedicated program.

Symbols can certainly be created and duplicated within generic drawing software but the process might be less efficient than using pencil and paper. This is therefore an area of notation where it is often best to avoid using ICT.

Back to index of teaching notation ideas

---

Idea 5: Use graphic and pupil-devised notations

Bear in mind that the Key Stage 3 curriculum specifies 'notations' in the plural and that pupils will have encountered graphic notations in Key Stage 2.

Getting pupils to devise their own notation symbols is a good way of getting them to think more deeply about the nature of sounds, textures and structures. They should already have used graphic notations in Key Stage 2. Getting them to discuss whether everybody can 'read' their graphic notations and produce sounds as they intended is a good way to introduce standard notation.

Some programs, such as Sibelius' Groovy Music series (www.sibelius.com/products/groovy), are designed around this kind of use of graphic notation and are also capable of representing the same sounds using standard notation. A program such as this might be a useful transitional tool across the key stages.

Back to index of teaching notation ideas

---

Idea 6: Use standard notation software wisely

Many programs focus on producing standard music notation of publication quality. These range from expensive professional software to freeware with limited features (see below for some examples). There is little doubt that, if standard notation is a suitable vehicle for the learning task in hand, notation software can have several advantages over hand-written methods:

• Nearly all programs are capable of playing back the music that is displayed. This develops critical listening skills as pupils check whether music they have entered gives the desired result.
• A visible playline moves across the score as the music plays, which makes it almost impossible for listeners to 'lose their place'.
• Making changes is quick and easy, e.g. changing a single pitch or transposing an entire phrase up an octave.
• It is easy to copy whole sections of music without having to re-enter every symbol.
• Results look neat. (Teachers should remember that this is always the case with notation software, even if musicality is poor. Any marks awarded for neatness have no bearing on musical ability.)
• Most programs allow realtime performance entry via MIDI, in time with a metronome click. The program will show what was played using standard notation. Potentially, this makes it possible for pupils to include chromaticism and syncopated or complex rhythms that they may not yet fully understand how to notate. (In practice, this sometimes produces unexpected notation unless appropriate settings are made.)

In many programs it is impossible to put the wrong number of beats in a bar – rests and ties are drawn automatically by the software. In other programs this is not the case and pupils need to be responsible for understanding and checking for mistakes themselves. Teachers should consider which is more desirable.

Teachers can use notation software to prepare their own teaching resources. Some programs offer free browser plugins so that notation files can be played interactively (but not edited) in a web browser. (See www.sibelius.com/scorch for more information about one such plugin.) The full notation software does not need to be installed on the computer to use the browser plugin. This opportunity is useful when preparing resources for a Learning Platform or a Virtual Learning Environment (VLE), which may need to be accessed from any network computer or from home.

Some programs make it quick and easy for teachers to generate traditional-style music notation worksheets for pupils. These can reinforce understanding of notation and other aspects of music theory. Many teachers use these as a standby for staff absence.

Before asking pupils to use a notation program in KS3, teachers should consider whether pupils are likely to be able to produce a musical product that they would be happy with. If not, their creativity would be restricted and they might become frustrated or disillusioned – a more appropriate vehicle should be chosen. With notation software, a carefully defined, realistic and musical pupil task is more likely to be appropriate in KS3 than a free composition. At KS4 and above there are stronger reasons to channel free creativity through notation software.

Some prominent scorewriting software contenders include Sibeliuswww.sibelius.com), the Finale family (www.finalemusic.com), which includes a free introductory version called NotePad, Notion (www.notionmusic.co.uk) and Personal Composer (www.personalcomposer.co.uk). You should search and explore websites to compare for yourself as there is simply too much information to summarise here. (

Programs such as Words and Music (www.topologika.com) are different in that they are designed specifically for use across the primary/secondary transition. It is possible to switch between standard and graphic notations and also to compare major and minor versions of the same tune. This program also has special features for adding lyrics.

Back to index of teaching notation ideas

---

Idea 7: Consider interactive notation activities

Several publishers sell collections of simple-to-use interactive applications that can be used with a whole class on an interactive whiteboard, perhaps as lesson starter activites. Some collections include skill development activities that are notation-based.

The interactive nature of ICT resources can encourage engagement through promoting a sense of ownership and competition. If pupils can easily make up their own notation patterns they can challenge each other to play what they have made and thereby improve their musical skills. Manipulating notation symbols in a computer grid is much more flexible than using traditional flashcards.

The linked activity below will launch a free demo version of an application to improve basic rhythm reading/performing skills. Users can devise their own rhythm patterns and show them either as intuitive simple blobs or as the equivalent standard notation.

from Interactive Whiteboard Activities for Music (Badger Publishing)

Using ICT offers the potential to flick quickly between the two and so strengthen association of symbols. The user's ability to rotate the grid presents an entirely new but related challenge that would be difficult to implement in the physical world. The success of this type of activity depends largely on how the teacher manages the classroom situation to set appropriate challenges and goals for pupils.

Herwig Weiser's electrochemical sound-modules

These are beautiful looking although, as a work in progress, I'm not really sure what their function is.

In any case, Herwig Weiser makes some arresting, sometimes highly conceptual, projects that use technology and, in some cases, sound. More via his site - http://www.zgodlocator.org/.

Silence in the words of...

A lift from the blog of Berlin sound artist Andreas Bick (http://silentlistening.wordpress.com/) which collects a number of quotes based around the concept of "silence".

Silence a boundary notion, a conceptual ideal, an aural vanishing point. Saying something about silence equals losing it. Being silent is impossible may it include to cease living. Many words have been uttered to grasp this paradox.

There is no such thing as silence. Something is always happening that makes a sound.

No Silence exists, that is not pregnant with sound.

Until I die there will be sounds. And they will continue following my death. - John Cage

Silence is a word which is not a word, and breathes an object which is not an object. - Georges Bataille

We should be sensitive to the thread of silence from which the tissue of speech is woven. - Maurice Merleau-Ponty

There are two silences. One when no word is spoken. The other when perhaps a torrent of language is being employed. This speech is speaking of a language locked beneath it. That is the continual reference. The speech we hear is an indication of that we don’t hear. It is a necessary avoidance, a sly, anguished or mocking smokescreen which keeps the other in its place. - Harold Pinter

Noise may have lost its power to offend. Silence hasn’t. - Dan Warburton

The fear of silence is nothing new. Silence surrounds the dark world of death. Sometimes the silence of the vast universe hovers over us, enveloping us. There ist he intense silence of birth, the quiet silence of one’s return to the earth. Hasn’t art been the human creature’s rebellion against silence? Poetry and music were born when man first uttered sound, resisting the silence. - Toru Takemitsu

When silence is conceived as the rejection of the human personality, the ulitmate silence is death. Then man likes to surround himself with sounds in order to nourish his fantasy of perpetual life. In Western society silence is negative, an embarrassment, a vacuum. Silence for Western man equals communication hangup. If one does not speak, the other will speak. - R. Murray Schafer

The essence of sound is felt in both motion and silence, it passes from existent to nonexistent. When there is no sound, it is said that there is no hearing, but that does not mean that hearing has lost its preparedness. Indeed, when there is no sound, hearing is most alert, and when there is sound the hearing nature is least developed. - Kirpal Singh

But if it’s true that all symphonies end in silence, it’s equally true that they begin there as well. Silence, after all, both buries and births us, and just as life without the counterweight of mortality would mean nothing, so silence alone, by offering itself as the eternal Other, makes our music possible. - Marc Slouka

Any given silence takes its identity as a stretch of time being perforated by sound. A genuine emptiness, a pure silence, are not feasible — either conceptually or in fact. If only because the art-work exists in a world furnished with many other things, the artist who creates silence or emptiness must produce something dialectical: a full void, an enriching emptiness, a resonating or eloquent silence. Silence remains, inescapably, a form of speech (in many instances, of complaint or indictment) and an element in a dialogue. - Susan Sontag

There is a beautiful page with lots of writings about silence. And please check the essay „The Aesthetics of Silence“ of Susan Sontag.

How Vision Affects Sound Processing

Source: http://musicalbionics.wordpress.com/

Publication date: January 18, 2010

There are a pair of papers in the most recent issue of Current Biology that got my attention thismorning.

There’s an original research article (Kayser, Logothetis, & Panzeri, 2010), and a handy review (Ghazanfar & Lemus, 2010). Full refs below.

There has been a flurry of recent research indicating that visual stimuli can influence activity in what were previously thought to be purely “auditory” areas of the brain – from the (relatively) high levels of the auditory cortex (Kayser, Petkov, & Logothetis, 2008) right down to the brainstem (Musacchia, Sams, Skoe, & Kraus, 2007). Interestingly, the effects of visual stimuli on activity in the brainstem seem to be even larger in musicians, but that’s a story for another time. It’s been mostly assumed, naturally enough, that the enhancements in behaviour (in the form of decreased reaction times, detection thresholds etc) are due to enhanced neural responses in the brain elicited by multisensory stimuli. Until recently, this assumption has never really been tested. Kayser et al (2010) tested this, by comparing the amount of stimulus-relevant information carried by neural responses in the auditory cortex of alert monkeys, who were watching videos of monkey vocalisations.

In addition to finding enhanced activity to multisensory stimuli, they also found an interesting relationship with the auditory-only responses. As is often found in these types of studies, the amount of enhancement gained from the multisensory stimuli increased as the auditory-only responses decreased. That is, the biggest gains occurred when the auditory signal alone was weakest. Strong auditory responses were actually suppressed by the addition of visual stimuli. At the same time, the information content of the auditory cortex activity increased. So, when congruent auditory-visual stimuli are presented, the effect of information from the visual system seems to selectively suppress strong auditory activity, which in turn reduces its variability, and increases its information content. Conversely, weak auditory signals are amplified by visual information, but at the expense of increased variability and lower information content.

Musacchia, G., Sams, M., Skoe, E., & Kraus, N. (2007). Musicians have enhanced subcortical auditory and audiovisual processing of speech and music. Proceedings of the National Academy of Sciences, 104(40), 15894-15898. doi: 10.1073/pnas.0701498104

Ghazanfar, A. A., & Lemus, L. (2010). Multisensory Integration: Vision Boosts Information through Suppression in Auditory Cortex. Current Biology, 20(1), R22-R23. doi: 10.1016/j.cub.2009.11.046

Kayser, C., Logothetis, N. K., & Panzeri, S. (2010). Visual Enhancement of the Information Representation in Auditory Cortex. Current Biology, 20(1), 19-24. doi: 10.1016/j.cub.2009.10.068

Kayser, C., Petkov, C., & Logothetis, N. (2008). Visual Modulation of Neurons in Auditory Cortex. Cereb Cortex. doi: 10.1093/cercor/bhm187

Sound and Geographic Visualization

Originally Published in Alan MacEachren and D.R.F. Taylor (eds.). 1994 Visualization in Modern Cartography. New York: Pergamon. pp. 149-166.

"Who the hell wants to hear actors talk?"
- Harry Warner on being confronted with the prospect of the sound movie.

Introduction

The issue of sound in the context of visualization may at first seem incongruous. There is, however, evidence to support the claim that sound is a viable means of representing and communicating information and can serve as a valuable addition to visual displays. Abstracted two-dimensional space and the visual variables - the traditional purview of cartography - may not always be adequate for meeting the visualization needs of geographers and other researchers interested in complex dynamic and multivariate phenomena. The current generation of computer hardware and software gives cartographers access to a broadened range of design options: three-dimensionality, time (animation), interactivity, and sound. Sound - used alone or in tandem with two-or three-dimensional abstract space, the visual variables, time, and interactivity - provides a means of expanding the representational repertoire of cartography and visualization.

This chapter discusses the use of realistic and abstract sound for geographic visualization applications. Examples of how and why sound may be useful are developed and discussed. Uses of sound in geographic visualization include sound as vocal narration, as a mimetic symbol, as a redundant variable, as a means of detecting anomalies, as a means of reducing visual distraction, as a cue to reordered data, as an alternative to visual patterns, as an alarm or monitor, as a means of adding non-visual data dimensions to interactive visual displays, and for representing locations in a sound space. The chapter concludes with research issues concerning sound and its use in geographic visualization.

Experiencing and Using Sound to Represent Data

Our sense of vision often seems much more dominant than our sense of hearing. Yet one only has to think about the everyday environment of sound surrounding us to realize that the sonic aspects of space have been undervalued in comparison to the visual (Ackerman 1990, Tuan 1993). Consider the experience of the visually impaired to appreciate the importance of sound and how it aids in understanding our environment. Also consider that human communication is primarily carried out via speech and that we commonly use audio cues in our day to day lives - from the honk of a car horn to the beep of a computer to the snarl of a angry dog as we approach it in the dark (Baecker and Buxton 1987).

There are several perspectives which can contribute to understanding the use of sound for representing data. Acoustic and psychological perspectives provide insights into the physiological and perceptual possibilities of hearing (Truax 1984, Handel 1989). An environmental or geographical perspective on sound can be used to examine our day to day experience with sound and to explore how such experiential sound can be applied to geographic visualization (Ohlson 1976, Schafer 1977, Schafer 1985, Porteous and Mastin 1985, Gaver 1988, Pocock 1989). Understanding how sound and music is used in non-western cultures may inform our understanding of communication with sound (Herzog 1945, Cowan 1948). Knowledge about music composition and perception provides a valuable perspective on the design and implementation of complicated, multivariate sound displays (Deutsch 1982). Many of these different perspectives have coalesced in the cross-disciplinary study of sound as a means of data representation, referred to as sonification, acoustic visualization, auditory display, and auditory data representation (Frysinger 1990). Within this context both realistic and abstract uses of sound are considered.

Using Realistic Sounds

Vocal narration is an obvious and important use of realistic sound. (note 2) Details about the physiological, perceptual, and cognitive aspects of speech are well known (Truax 1984, Handel 1989) and film studies offer insights into the nature and application of vocal narration (Stam, Burgoyne, and Flitterman-Lewis 1992).

Another use of realistic sounds is as mimetic sound icons, or "earcons" (Gaver 1986, Gaver 1988, Gaver 1989, Blattner et al. 1989, Mountfort and Gaver 1990). Earcons are sounds which resemble experiential sound. Gaver, for example, has developed an interface addition for the Macintosh computer which uses earcons. An example of an earcon is a "thunk" sound when a document is successfully dragged into the trash can in the computer interface.

Using Abstract Sounds

Abstract sounds can be used as cues to alert or direct the attention of users or can be mapped to actual data. Early experiments by Pollack and Ficks (1954) were successful in revealing the ability of sound to represent multivariate data. Yeung (1980) investigated sound as a means of representing the multivariate data common in chemistry after finding few graphic methods suitable for displaying his data. He designed an experiment in which seven chemical variables were matched with seven variables of sound: two with pitch, one each with loudness, damping, direction, duration, and rest (silence between sounds). His test subjects (professional chemists) were able to understand the different patterns of the sound representations and correctly classify the chemicals with a 90% accuracy rate before training and a 98% accuracy rate after training. Yeung's study is important in that it reveals how motivated expert users can easily adapt to complex sonic displays.

Bly ran three discriminant analysis experiments using sound and graphics to represent multivariate, time-varying, and logarithmic data (Bly 1982a). In the first experiment she presented subjects with two sets of multivariate data represented with different variables of sound (pitch, volume, duration, attack, waveshape, and two harmonics) and asked subjects to classify a third, unknown set of data as being similar to either the first or second original data set. The test subjects were able to successfully classify the sound sets. In a second part of the experiment she tested three groups in a similar manner but compared the relative accuracy of classification among sound presentation only (64.5%), graphic presentation only (62%), and a combination of sound and graphic presentation (69%). She concluded that sound is a viable means of representing multivariate, time-varying, and logarithmic data - especially in tandem with graphic displays.

Mezrich, Frysinger, and Slivjanovski confronted the problem of representing multi-variable, time-series data by looking to sound and dynamic graphics (Mezrich et al. 1984). They had little success finding the graphic means to deal with eight-variable time series data. An experiment was performed where subjects were presented with separated static graphs, static graphs stacked atop each other (small multiples), overlaid static graphs, and redundant dynamic visual and sound (pitch) graphs. The combination of dynamic visual and sound representation was found to be the most successful of the four methods.

An ongoing project at the University of Massachusetts at Lowell seeks to expand the use of sound for representing multivariate and multidimensional data. The "Exvis" project uses a one-, two-, and three-dimensional sound space to represent data (Smith and Williams 1989, Smith et al. 1990, Williams et al. 1990, Smith et al. 1991). The project is based upon the idea of an icon: "an auditory and graphical unit that represents one record of a database" (Williams et al. 1990, 44). The visual attributes of the icon are "stick-figures" which can vary in "length, width, angle, and color" (Williams et al. 1990, 45). The sonic attributes of the icons are "pitch, attack rate, decay rate, volume, and depth of frequency modulation" (Williams et al. 1990, 45). An experimental Exvis workstation has been set up to run various human factors experiments, and initial tests of subjects have been completed. The results reveal that using visual and sonic textures together improves performance.

Two-dimensional sound displays which locate sounds up/down, right/left via stereo technology, and three-dimensional sound displays which add front/back to two dimensional displays are also being developed. A three-dimensional virtual sound environment has been developed at the NASA-Ames Research Center (Wenzel et al. 1988a, Wenzel et al. 1988b, Wenzel et al. 1990). The ability to locate sound in a multidimensional "sound space" will undoubtedly be important for representing spatial relationships...

More here.

Augmented Reality drumkit

Augmented Reality Drum Kit (demo #1) from squidder on Vimeo.

"So AR seems to be the 'next big thing' for the year ahead...and the folks over at Squidder.com have combined it with fiducial tracking to make a demo of an AR drumkit. Air drums anyone?"

Lifted from this blog.

Record grooves under an electron microscope

Found at this link: here

Chris Supranowitz is a researcher at The Insitute of Optics at the University of Rochester. Along with a number of other spectacular studies (such as quantum optics, trapping of atoms, dark states and entanglement), Chris has decided to look at the relatively boring grooves of a vinyl record using the institute’s electron microscope. Well, not boring for me.

From what I read, it’s not just a simple matter of sticking a record under a fancy microscope, as there is a lot of preparation (such as gold-sputtering the surface) and post-processing to be done. Having said that, the results are very cool:

Here’s a single groove even closer still, magnified 1000 times:

Chris also did the pits in a CD – here’s what they look like, just for contrast:

Chris decided to take the whole electron microscope image one step further, and created a blue/red 3-dimensional image of the record groove! So, if you have a pair of 3D glasses (sorry, the ones you got from watching Avatar won’t work – you need red on the left, blue on the right), throw them on and take a look at this amazing picture:

Maybe these vinyl grooves are only beautiful to an audio geek like me, but I think that these images are truly spectacular. Thanks to noiseforairports for the tip.

Comments »

• Obbop says:

  February 17, 2010 at 3:31 pm

  With enough practice, just as a maestro can view sheet music and hear a symphony play the music in his mind, an engineer can scan the vinyl grooves and hear the song inside.

  Admittedly, it IS more difficult doing so viewing a quadraphonic vinyl record.
• Doug McArthur says:

  February 17, 2010 at 4:25 pm

  That is so neat. You can actually SEE the waveforms cut into the record. The term “cut a record” has so much more meaning at this level of detail.

Spectrogram #1

Ever the maverick, electronica fiend Aphex Twin made a track in 1999 that is often referred to as 'Formula', 'Equation' or 'Mathematical Equation' [its actual full title is the long-winded 'ΔMi−1=−αΣn=1NDi[n][Σj∈C{i}Fji[n − 1]+Fexti[(n−1)]'.]

The interesting part which specifically relates to visualizing music is that it was embedded with sound that would create an image when re-played as a spectrogram. At 5:30 on the above video you can see the appearance of a face - an image of Richard 'Aphex Twin' James.

New sound recorder

I also picked up a new piece of hardware today. Previously I've been using my dinky Olympus digital voice recorder but now I have this: the Zoom H4 stereo/multi-track recorder which records both wav and mp3 formats and comes supplied with Cubase LE4 software. And input for guitar and bass - although I can't see this function getting much use.

Anyway, it's a bit of a beast by comparison with an appearance somewhere between a Taser and a prop from Battlestar Galactica. I'm actually too frightened to switch it on right now.

Source Book of Proposed Music Notation Reforms by Gardner Read

An interesting find in the library this afternoon was Gardner Read's 'Source Book of Proposed Music Notation Reforms'. It basically catalogues the attempts to usurp traditional music notation over the years and some of the examples prove to be interesting visual works. Many address specific problems that have arisen in denoting particular musical elements but none really have had the ability to replace the classic Western format that most of us will be familiar with. A lot, however, seem to be very similar variations. At least to this musically untrained eye.

There may be the potential to incorporate some of these into descriptions of noise - particularly those symbols that represent the length and pitch of a particular sound. Notation for percussion instruments might also be of use given how it is less about representing melody.

Visual Complexity inspiration

Pop Sketch Series / Jake Elliott The Shape of Song / Martin Wattenberg Narratives 2.0 / Matthias Dittrich Visualization of iTunes Libraries / Caleb Larsen 3D Tour of Classical Music History / Anonymous Prof Mo Money Mo Problems / Nick Hardeman Chart Arcs / Martin Dittus More at www.visualcomplexity.com

First group tutorial

Today was our first group tutorial of the semester with Tash. I'm already seeing some potential overlaps between the kind of areas we will individually be researching. Plus I'm once again inspired by the rest of the group's projects.

Anyway, the key thing today was in deciding to put the 'book design' aspect on the back burner for a while. It's certainly something that I'm interested in and it may very well inform the aesthetic choices I make later on, but - for now, at least - I need to concentrate on the first stage of my research which will be looking at existing ways of visualizing sound. I've now got together a long list of chapters (which can be edited down) and I really want to get this part of the project completed fully before I move on. I'll also be doing some documenting of individual libraries but I won't quite know what I'm doing with the material until I figure out what potential there is in representing it.

Tomorrow morning I'll be mentoring: doing some individual tutorials with design studies undergraduates (which I hope will also prove inspiring). Then I'll be back in the library. Given the amount of reading up I still have to do, it's probably for the best that this project is based around libraries.

More sound visualizations

More Youtube featured experiments in visualizing sound.

Participation proposal submitted

I submitted my abstract in response to the call for participants to the organisers of the Bigger Than Words conference today. I've attached the design for the cover.

While initially very similar, the key difference between this and what I'd already sent to Lawrence and Tash was that I didn't really mention anything about book design this time: I thought that the notes on aesthetics and additional design influences were perhaps less important within the context of the conference.

What I did emphasise was the importance of classification to this study and also referenced the relevance of both Wassily Kandinsky and John Cage to the subject of visualising sound. I'd also read up a little on Spectrograms so these got a mention too.

All this is proving helpful. I've decided to get the first part of the report finished early for this semester and this reworking of my proposal in its various forms is assisting in the devising of the report's chapters. This week I want to get the 'Contents' close to finalised so that I can start the subject research in earnest.

Ligeti - Artikulation

Another interesting, very 'Eames era' visual representation of sound.

Death of the Diner by Ed Nacional

More work by Ed Nacional. I had already mentioned to Dave about an idea for a folded book/brochure for this project. (That idea is currently scrawled on a scrap of paper somewhere.) Meantime, I spotted this - titled 'The Death of the Diner' - which itself is a study in typology with a nice typographic touch to make the project completely cohesive.