*blog... kind of... *rss 



svg tag + audio tag = 3D Waveform


As always, you just need a bit of practice with a language to start using it in nice ways. Now that I had that little 3D engine working and with the <audio> around, it was time to produce an idea I always had in mine. A 3D interpretation of a waveform.

I'm sure the first thing you would think after checking this experiment would be... What? I didn't know I could analyse the sound signal with the <audio> tag?! ... Well, you can't, if you check the source code (*hint* right click -> view source) you'll see an array of numbers. These are all the sound level values of the waveform at 30 fps.

I got these values using the library BASS for linux. Unfortunately, my C skills aren't so good (example) and I can't seem to control how to get the values exactly at the fps I want without getting desyncronisations. The first part of the visualisation is spot on, but by the end things aren't that impressive. I'll keep researching on this and update a new array of values whenever I crack it.

Dean McNamee showed me the way. Forget C and BASS. All you need is python, the tune in .wav and 23 lines of magic.

# python script.py audio.wav > output.txt

import math
import struct
import wave
import sys

w = wave.open(sys.argv[1], 'rb')
# We assume 44.1k @ 16-bit, can test with getframerate() and getsampwidth().
sum = 0
value = 0;
amps = [ ]

for i in xrange(0, w.getnframes()):
	# Assume stereo, mix the channels.
	data = struct.unpack('<hh', w.readframes(1))
	sum += (data[0]*data[0] + data[1]*data[1]) / 2
	# 44100 / 30 = 1470
	if (i != 0 and (i % 1470) == 0):
		value = int(math.sqrt(sum / 1470.0) / 10)
		amps.append(value)
		sum = 0

print amps

I've updated the experiment with the new values; now is perfectly in sync all the time. Which makes you appreciate the little sounds at the end much more. Thanks Dean!

As you'll see on the source code, the rest is very simple. Create a couple of cubes, place them one in from on each other, and modify their scaleY depending on the waveform they are related to in that step.

The end result is quite interesting I think and I hope doing more like these. Hopefully more interactive next time.

The factor that turned this from just a nice experiment to an awesome was the fact that the eedl guys allowed me to use their (great) tunes for my experimentation.

Thanks once again!
23 comments

Hi!
It exist an other solution: using video part for audio spectrum data
http://www.tapper-ware.net/devel/js/JS.VideoViz/index.xhtml

;)
Uh, how do you make that work? I've tried on Firefox/Chrome/Chromium and the visualisation box is always empty, music plays tho.
Cool experiment!
I heard EchoNest works fairly well for sound analysis stuff, you can submit your track online and get back an xml file of the sound data. Might fix your framerate issue, no need for C !
http://www.echonest.com

- Karl
Nah, it is easier than that. Someone has already told me the proper way (python + .wav). Will update later.
@mr.doob: That work on my FF3.5.5 (PC & Mac), but not on webkit (maybe not supported yet)

https://developer.mozilla.org/En/manipulating_video_using_canvas
Doesn't seem to work on FF3.5.4 (Linux) :(
get an error when trying to run the script:

Traceback (most recent call last):
File "/Users/elkraneo/Desktop/test_wav/output.py", line 6, in
w = wave.open(sys.argv[1], 'rb')
IndexError: list index out of range

macosx Python 2.6.4 ...any idea ?
Uh, no idea to be honest. Maybe you're not using it properly... I use it like this... python script.py audio.wav > output.txt
thx, tried that way and get
File "output.py", line 15, in
data = struct.unpack(' struct.error: unpack requires a string argument of length 4

:(...checking the encoding of the .wav
i get the same error elkraneo gets:

Traceback (most recent call last):
File "/Users/elkraneo/Desktop/test_wav/output.py", line 6, in
w = wave.open(sys.argv[1], 'rb')
IndexError: list index out of range

I believe this is because the audio file is not of correct format. Even if it is .wav, there are different ways wavs can be encoded.

mr doob, do you know of one that would work to decode mp3s or be more flexible in terms of getting the audio data for more input media types?
oh man, this is fucking awesome!!
I dont know python, is there anything to use audio frequencies in some way? I wanna try it too

Drawing it in circles would be nice.
Great engine ! So many variations are possible... I'll try it too !

The code is awesome. Here's a variation : http://miniloop.bcolin.com/xp1/

Thanks again !
Nice one! :D
damn
Impressive :)

I love the song too
Hey.

Love this.

I need some documentation to try this experiment myself. How do I use python lol. I can paste above code in my Cl python or gui I do not know how to define the wav and define the output path..I am totally noob and need some pointers.

Great work..

Thanks.
I think you can google most of that stuff ;)
In Firefox 4 you can analyze and visualize audio on the fly, see http://blog.mozilla.com/blog/2010/09/07/firefox-4-beta-with-faster-graphics-and-new-audio-capabilities-for-the-web/
as usual wasnt careful a poor encoding choice earlier on in ogg/vlc made a junk file.
Awesome! You have the best OpenGL demos on the web!
This is the first time when I'm impressed by some OpenGL experiments.

Your framework Three.js rules, I can't wait to experiment with it during this weekend. Thank you for making it available open-source.

Alex