ricardocabellov5.5

I think it was about one year ago, right after the Google Sphere experiment, when Aaron started talking about a music video in the browser idea and if I would be up for it. If you consider my background, you'll understand how excited I was with such opportunity. Since then they (the Google Creative Lab guys) started looking for a band to work with and a director. Months later we got busy with The Johnny Cash Project and seeing how well it worked out they saw clear how Chris could do a great job directing this project too. Chris happened to be friends with Arcade Fire whom also seemed to be interested on joining the party.

As the project was considerably big we also needed a production company to handle the design and development process and B-Reel seemed a good fit. Albeit they had some in-house developers we really needed people with HTML5/Javascript experience. Finding people with these skills turned out to be a hard task as it seems most of the people we knew were stuck with either jQuery-kind-of-Javascript, AS3 or full employed/unavailable... As an act of desperation I tweeted this. An old friend of mine replied to the call. I knew he had already played around with <canvas> and I knew he would be invaluable for the team for his know-how. At this point, we had a band, a track, Chris+Aaron had the script ready and the team was all set.

At first I didn't realised the cleverness of the idea of using Google Maps/Street View data set. It wasn't until we had the first test of the kid running around the neighbourhood that it made an impact on me and made me remember old times. Kudos to Chris and Aaron for envisioning that :)

Production time

By reading the script I realised that how valuable the javascript libraries I had been developing for the past year were going to be. We had a sequencer ready to add and remove effects in sync with a tune, a 3D renderer and Harmony (as it was referred in the script).

I'm sure most of the people will think that the drawing tool is basically what I did... not really. Although it used Harmony as the base code, George was in charge of that part. He did a great work by creating a new brush out of it, a recorder and repainter and, in the last minute, some keyboard based input with the letter being drawn using that brush.

Eduard was responsible of the whole mainframe and making sure everyone was producing compatible code. The sequencer already provided some basic template for show/hide behaviour of effects but we also needed pre-loading, sharing data between effects and more things I'm probably not aware of. If that wasn't enough, he created a sequencing tool (with javascript) so the Director and Art Director could easily set when each window and effect would appear and where in the screen.

Jaime took the maps beast and geocoding utils. We couldn't just directly use the embeddable Google Maps, the maps were supposed to have some tilting on the camera so a new Maps Data viewer was required. We also had to figure out possible routes for the runner to get to the user's home in sync with the music, the Direction Route API provided that but we had to implement it properly. All this takes a lot of time, research and testing.

I was going to work on all the Street View scenes and also the cgi version of the runner. However, I didn't have skinning nor any animation code for three.js yet, so as these scenes weren't interactive neither customisable it was quickly decided that B-Reel would create videos for these parts — which ended up really great too! In the end I also added the birds flocking and birds landing on the drawing to my plate.

Now... I can't speak much about the challenges that other people faced but you can get the idea with the ones I did.

(Fast) Street View

At first we intended to simply use Google's Street View. I did a test integrating three.js with it and it seemed to work all fluid. However, what I didn't know was that if you have WebGL enabled Google's Street View would already use it. So what other people would see was considerable slower than what I was seeing. If WebGL is not enabled the Street View uses a three.js-like renderer. That was fine on Windows and Linux but not so much for MacOS. Turns out Google Chrome internally uses a different graphics library in MacOS than in Windows and Linux. CoreGraphics for MacOS and Skia for Windows and Linux. Each library have their own pros and cons but CoreGraphics is specially slow when transforming and clipping big images. Street View would run at 30fps on Windows/Linux while it would get 1fps on MacOS.

Like with the maps, we had to build a custom Street View Data viewer. Jaime encountered the same problem while doing the 3d maps using three.js. He then started researching other ways of drawing the Maps Data in a way that would create the same effect. An additional challenge was that with <canvas> you don't have access to the pixel data of images loaded from another domain. Otherwise we could just use this technique and call it a day. However, albeit pixel access is forbidden, context.drawImage() is allowed for copying areas from images hosted on other domains.

By stitching all the tiles the API provides for each panorama we get this image:



After zooming in to a part of the image we get this:



Somehow we need to apply this distortion:



We can do that by cropping columns from the original image and positioning them one after the other horizontally. Each one with some vertical scaling depending of the proximity to the center. We get this:



Now we just need to wider the columns a bit to hide the gaps:



The distortion isn't perfect but it's close enough. This approach seemed to be quite fast in all the platforms and all it was left was to apply the good old Perlin Noise to the movement to get some human feeling.

The right heading

Or so I thought. We were missing an important bit. For each StreetView we had to place the camera target at specific positions. For instance, the Street View that does the 360 right in front of your house had to start spinning right from your house. But how do you know where to look at? Where is the user's house? The Street View Service doesn't give any information about that. After studying all the data the API provided and directly debugging Google Maps I ended up noticing that each panorama has a lat/lng information, plus I also had the lat/lng information of the location of the house. On top of that, the panorama does provide the angle where the north points to.

Very long story short... subtract the lat/lng position of the house with the lat/lng position of the panorama. Get the angle of that vector and mix it with the angle of where the north is in the panorama. Voila! :)

Birds



Although Guille and Michael had done great progress with the birds we felt we could do better. After considering the options, my approach was using a 3 polygons mesh — one polygon for each wing and one for the body. Then animating it by sinusoidally moving up and down the vertices at the end of the wings. Although it didn't look like a crow it gave, once again, a close enough effect. Specially when you have a bunch of them following a boid simulation.

Tinting

This one is going to be controversial... Street View and Maps footage needed to be colour corrected because the action is supposed to take place in the morning, thus some yellowish tint was needed.



Again, we can't access the pixel data of images hosted in another domain, so the only option was to layer a colour on top of the image and play with blend modes. However, take a look at the blend modes available... only lighten could be of some use here. So we first tried drawing a rectangle on top with a yellow colour and lighten blending mode enabled. That kind of worked but it washed the footage out.



There is another blending mode though... darken. It was taken out of the specification but it still remains in WebKit and I hope they put it back because this is a good example of why is useful. By drawing that yellow colour using the darken blending mode on top of the image, and then drawing the original image on top using the lighten blending mode we achieved a really nice yellow tint and contrast that worked quite well for simulating the light we get in the morning.



Notice how the darks stay dark. Here it's the actual snippet:

var context = texture_mod.getContext( '2d' );
 
context.drawImage( texture, 0, 0 );
 
context.globalAlpha = 0.5;
context.globalCompositeOperation = 'darker';
context.fillStyle = '#704214';
context.fillRect( 0, 0, TEXTURE_WIDTH, TEXTURE_HEIGHT );
 
context.globalCompositeOperation = 'lighter';
context.drawImage( texture, 0, 0 );

Tween and Manual Tween

Most of the movements in the video use the well known Penner's Easing equations.

Sole had been working on a simplified Javascript Tweening library some months before this project and it proved really useful. You probably know how Tweening libraries work... you define an animation with its properties to be animated, it's delay and so on... then you start it. However, we needed to be able to go backwards at any point of the video (at least I needed it ;P). So by doing the Manual Tween alternative we were able to move freely to any point of the virtual timeline.

Now that I look back, instead of having 2 different libraries for Tweening, the Manual one should have been part of the sequencer code... hmmm... something to consider...

Optimising

Launch day was approaching and the server guys were giving some recommendations on some changes we could do for making the server happier. One of them was the tree animation I was using on the last Street View part. It was something I intended to do but I didn't have time just yet. Instead of having 63 separate images for a growing tree animation is better to mix them up in a single image.



We have seen this in previous projects haven't we? ;)

The second one was about javascript files.... we had about 40+ javascript files. The more files you have the more server resources each user will consume. There is a defined amount of connections available in a server, if that amount is, for example, 40 and each user needs to open 40 files to be able to see the website, the website would be able to be seen by just 1 user at a time. Mix all those 40 files into 1 file and then 40 users will be able to visit the website at a time. We didn't used the compressed/compiled index by launch time and I believe that was one of the main reasons we suffered some downtimes.

This script shows pretty much how to combine and minify easily.

Ok, that's enough

I know, that was quite long wasn't it. Hope this is of some use to some one, and hope you liked the actual piece too. Now, let's move to WebGL ;)

PS: If you wonder about any other technical details, feel free to use the comments and I'll try to address them.