ricardocabellov5.5

I've been asked this question a few times for the past week so I thought I would make a "public" answer.

At first I tested some IDEs like NetBeans and Aptana but somehow I didn't get to like them. Specially because they tend to add "hidden" folders around.

Believe it or not, I simply use Ubuntu's default text editor gedit. Sometimes the syntax highlighting is not correct and editing 1 line files is incredibly slow but apart from that it doesn't get on the way. Simple and fast.

However, auto completion is something I was missing from my FDT/AS3 days. Turns out auto completion based on already written text in the loaded files is all I need.

I think it was about one year ago, right after the Google Sphere experiment, when Aaron started talking about a music video in the browser idea and if I would be up for it. If you consider my background, you'll understand how excited I was with such opportunity. Since then they (the Google Creative Lab guys) started looking for a band to work with and a director. Months later we got busy with The Johnny Cash Project and seeing how well it worked out they saw clear how Chris could do a great job directing this project too. Chris happened to be friends with Arcade Fire whom also seemed to be interested on joining the party.

As the project was considerably big we also needed a production company to handle the design and development process and B-Reel seemed a good fit. Albeit they had some in-house developers we really needed people with HTML5/Javascript experience. Finding people with these skills turned out to be a hard task as it seems most of the people we knew were stuck with either jQuery-kind-of-Javascript, AS3 or full employed/unavailable... As an act of desperation I tweeted this. An old friend of mine replied to the call. I knew he had already played around with <canvas> and I knew he would be invaluable for the team for his know-how. At this point, we had a band, a track, Chris+Aaron had the script ready and the team was all set.

At first I didn't realised the cleverness of the idea of using Google Maps/Street View data set. It wasn't until we had the first test of the kid running around the neighbourhood that it made an impact on me and made me remember old times. Kudos to Chris and Aaron for envisioning that :)

Production time

By reading the script I realised that how valuable the javascript libraries I had been developing for the past year were going to be. We had a sequencer ready to add and remove effects in sync with a tune, a 3D renderer and Harmony (as it was referred in the script).

I'm sure most of the people will think that the drawing tool is basically what I did... not really. Although it used Harmony as the base code, George was in charge of that part. He did a great work by creating a new brush out of it, a recorder and repainter and, in the last minute, some keyboard based input with the letter being drawn using that brush.

Eduard was responsible of the whole mainframe and making sure everyone was producing compatible code. The sequencer already provided some basic template for show/hide behaviour of effects but we also needed pre-loading, sharing data between effects and more things I'm probably not aware of. If that wasn't enough, he created a sequencing tool (with javascript) so the Director and Art Director could easily set when each window and effect would appear and where in the screen.

Jaime took the maps beast and geocoding utils. We couldn't just directly use the embeddable Google Maps, the maps were supposed to have some tilting on the camera so a new Maps Data viewer was required. We also had to figure out possible routes for the runner to get to the user's home in sync with the music, the Direction Route API provided that but we had to implement it properly. All this takes a lot of time, research and testing.

I was going to work on all the Street View scenes and also the cgi version of the runner. However, I didn't have skinning nor any animation code for three.js yet, so as these scenes weren't interactive neither customisable it was quickly decided that B-Reel would create videos for these parts — which ended up really great too! In the end I also added the birds flocking and birds landing on the drawing to my plate.

Now... I can't speak much about the challenges that other people faced but you can get the idea with the ones I did.

(Fast) Street View

At first we intended to simply use Google's Street View. I did a test integrating three.js with it and it seemed to work all fluid. However, what I didn't know was that if you have WebGL enabled Google's Street View would already use it. So what other people would see was considerable slower than what I was seeing. If WebGL is not enabled the Street View uses a three.js-like renderer. That was fine on Windows and Linux but not so much for MacOS. Turns out Google Chrome internally uses a different graphics library in MacOS than in Windows and Linux. CoreGraphics for MacOS and Skia for Windows and Linux. Each library have their own pros and cons but CoreGraphics is specially slow when transforming and clipping big images. Street View would run at 30fps on Windows/Linux while it would get 1fps on MacOS.

Like with the maps, we had to build a custom Street View Data viewer. Jaime encountered the same problem while doing the 3d maps using three.js. He then started researching other ways of drawing the Maps Data in a way that would create the same effect. An additional challenge was that with <canvas> you don't have access to the pixel data of images loaded from another domain. Otherwise we could just use this technique and call it a day. However, albeit pixel access is forbidden, context.drawImage() is allowed for copying areas from images hosted on other domains.

By stitching all the tiles the API provides for each panorama we get this image:



After zooming in to a part of the image we get this:



Somehow we need to apply this distortion:



We can do that by cropping columns from the original image and positioning them one after the other horizontally. Each one with some vertical scaling depending of the proximity to the center. We get this:



Now we just need to wider the columns a bit to hide the gaps:



The distortion isn't perfect but it's close enough. This approach seemed to be quite fast in all the platforms and all it was left was to apply the good old Perlin Noise to the movement to get some human feeling.

The right heading

Or so I thought. We were missing an important bit. For each StreetView we had to place the camera target at specific positions. For instance, the Street View that does the 360 right in front of your house had to start spinning right from your house. But how do you know where to look at? Where is the user's house? The Street View Service doesn't give any information about that. After studying all the data the API provided and directly debugging Google Maps I ended up noticing that each panorama has a lat/lng information, plus I also had the lat/lng information of the location of the house. On top of that, the panorama does provide the angle where the north points to.

Very long story short... subtract the lat/lng position of the house with the lat/lng position of the panorama. Get the angle of that vector and mix it with the angle of where the north is in the panorama. Voila! :)

Birds



Although Guille and Michael had done great progress with the birds we felt we could do better. After considering the options, my approach was using a 3 polygons mesh — one polygon for each wing and one for the body. Then animating it by sinusoidally moving up and down the vertices at the end of the wings. Although it didn't look like a crow it gave, once again, a close enough effect. Specially when you have a bunch of them following a boid simulation.

Tinting

This one is going to be controversial... Street View and Maps footage needed to be colour corrected because the action is supposed to take place in the morning, thus some yellowish tint was needed.



Again, we can't access the pixel data of images hosted in another domain, so the only option was to layer a colour on top of the image and play with blend modes. However, take a look at the blend modes available... only lighten could be of some use here. So we first tried drawing a rectangle on top with a yellow colour and lighten blending mode enabled. That kind of worked but it washed the footage out.



There is another blending mode though... darken. It was taken out of the specification but it still remains in WebKit and I hope they put it back because this is a good example of why is useful. By drawing that yellow colour using the darken blending mode on top of the image, and then drawing the original image on top using the lighten blending mode we achieved a really nice yellow tint and contrast that worked quite well for simulating the light we get in the morning.



Notice how the darks stay dark. Here it's the actual snippet:

var context = texture_mod.getContext( '2d' );
 
context.drawImage( texture, 0, 0 );
 
context.globalAlpha = 0.5;
context.globalCompositeOperation = 'darker';
context.fillStyle = '#704214';
context.fillRect( 0, 0, TEXTURE_WIDTH, TEXTURE_HEIGHT );
 
context.globalCompositeOperation = 'lighter';
context.drawImage( texture, 0, 0 );

Tween and Manual Tween

Most of the movements in the video use the well known Penner's Easing equations.

Sole had been working on a simplified Javascript Tweening library some months before this project and it proved really useful. You probably know how Tweening libraries work... you define an animation with its properties to be animated, it's delay and so on... then you start it. However, we needed to be able to go backwards at any point of the video (at least I needed it ;P). So by doing the Manual Tween alternative we were able to move freely to any point of the virtual timeline.

Now that I look back, instead of having 2 different libraries for Tweening, the Manual one should have been part of the sequencer code... hmmm... something to consider...

Optimising

Launch day was approaching and the server guys were giving some recommendations on some changes we could do for making the server happier. One of them was the tree animation I was using on the last Street View part. It was something I intended to do but I didn't have time just yet. Instead of having 63 separate images for a growing tree animation is better to mix them up in a single image.



We have seen this in previous projects haven't we? ;)

The second one was about javascript files.... we had about 40+ javascript files. The more files you have the more server resources each user will consume. There is a defined amount of connections available in a server, if that amount is, for example, 40 and each user needs to open 40 files to be able to see the website, the website would be able to be seen by just 1 user at a time. Mix all those 40 files into 1 file and then 40 users will be able to visit the website at a time. We didn't used the compressed/compiled index by launch time and I believe that was one of the main reasons we suffered some downtimes.

This script shows pretty much how to combine and minify easily.

Ok, that's enough

I know, that was quite long wasn't it. Hope this is of some use to some one, and hope you liked the actual piece too. Now, let's move to WebGL ;)

PS: If you wonder about any other technical details, feel free to use the comments and I'll try to address them.

In case you were wondering... and man... it feels good. So much space in my head now :)

Last friday MIX Online and An Event Apart launched an unusual Javascript contest. 10K Apart pushes the developer to reduce their code so it fits in 10,000 bytes. This sounds like a nice challenge, but strangely they allow specific external libraries, which, in my opinion, over complicate things.

However, just to see what could be done, I quickly checked the amount of bytes that I would need to have a simple and easy to use 3d engine. That turned out to be about 1,000 bytes. Which left me with 9,000 bytes. Now, this may sound great, but it's kind of discouraging. Now I understand why the 64 kbytes coders always say that 64 kbytes are way harder to do than 4 kbytes – the number of possibilities is lower. Anyway, if I come up with a good idea I may try doing something with that for the contest.

Today I found out about JS1k. Which seemed to be Peter van der Zee's response to the former contest (no external libs, just 1,024 bytes of plain javascript) and it also had a few submissions.

Once I got the 3d engine working I got a bit code-addicted and ended up doing a plasma in 3D in 1,464 bytes, it was nice looking already, so it was a matter of reducing those extra 440 bytes. After learning some tricks and testing things here and there, it got reduced to 996 bytes. Here it's the result:



Looking forward to see what p01 has to "say" of all this... :)

EDIT: After reading Diego's post and finding interesting to see which tricks he used I thought I should also share the non-obfuscated code of my entry.

( function () {

	var res = 25, res3 = res * res * res,
	i = 0, x = 0, y = 0, z = 0, s, size, sizeHalf,
	vx, vy, vz, rsx, rcx, rsy, rcy, rsz, rcz,
	xy, xz, yx, yz, zx, zy,
	cx = 0, cy = 0, cz = 1, rx = 1, ry = 1, rz = 0,
	t, t1, t2, t3,
	sin = Math.sin, cos = Math.cos, pi = Math.PI * 3,
	mouseX = 0, mouseY = 0, color,
	doc = document, body = doc.body,
	canvas, context, mesh = [],
	width = innerWidth,
	height = innerHeight,
	widthHalf = width / 2,
	heightHalf = height / 2;

	body.style.margin = '0px';
	body.style.overflow = 'hidden';

	canvas = doc.body.children[0];
	canvas.width = width;
	canvas.height = height;

	context = canvas.getContext( '2d' );
	context.translate( widthHalf, heightHalf );

	doc.onmousemove = function ( event ) {

		mouseX = ( event.clientX - widthHalf ) / 1000;
		mouseY = ( event.clientY + heightHalf ) / 1000;

	};

	while ( i++ < res3 ) {

		mesh.push( x / res - 0.5 );
		mesh.push( y / res - 0.5 );
		mesh.push( z / res - 0.5 );

		z = i % res;
		y = !z ? ++y %res : y;
		x = !z && !y ? ++x : x;

	}

	setInterval( function () {

		context.clearRect( - widthHalf, - heightHalf, width, height );

		cx += ( mouseX - cx ) / 10;
		cz += ( mouseY - cz ) / 10;

		t = new Date().getTime();
		t1 = sin( t / 20000 ) * pi;
		t2 = sin( t / 10000 ) * pi;
		t3 = sin( t / 15000 ) * pi;

		rx = t / 10000;

		rsx = sin( rx ); rcx = cos( rx );
		rsy = sin( ry ); rcy = cos( ry );
		rsz = sin( rz ); rcz = cos( rz );

		i = 0;

		while ( ( i += 3 ) < res3 * 3 ) {

			x = mesh[ i ];
			y = mesh[ i + 1 ];
			z = mesh[ i + 2 ];
			s = sin( t1 + x * t1 ) + sin( t2 + y * t2 ) + sin( t3 + z * t3 );

			if ( s >= 0 ) {

				xy = rcx * y - rsx * z;
				xz = rsx * y + rcx * z;

				yz = rcy * xz - rsy * x;
				yx = rsy * xz + rcy * x;

				zx = rcz * yx - rsz * xy;
				zy = rsz * yx + rcz * xy;

				vx = zx - cx;
				vy = zy - cy;
				vz = yz + cz;

				if ( vz > 0 ) {

					color = ( 64 / vz ) >> 0;
					context.fillStyle = 'rgb('+ ( color - 16 ) + ','+ ( color * 2 - 128 ) + ','+ ( color + 64 ) + ')';

					size = s * 30 / vz;
					sizeHalf = size / 2;

					context.fillRect( ( vx / vz ) * widthHalf - sizeHalf, ( vy / vz ) * widthHalf - sizeHalf, size, size );

				}

			}
		}

	}, 16 );

} )();

Which, after compression ends up like this:

var O=24,d=O*O*O,X=0,U=0,T=0,S=0,W,j,L,o,m,k,b,q,ac,n,ab,l,K,I,r,p,Z,Y,C=0,A=0,w=1,G=1,F=1,E=0,V,P,N,M,u=Math.sin,f=Math.cos,v=Math.PI*3,R=0,Q=0,H,g=document,D=g.body,h,aa,B=[],a=innerWidth,e=innerHeight,J=a/2,c=e/2;D.style.margin="0px";D.style.overflow="hidden";h=g.body.children[0];h.width=a;h.height=e;aa=h.getContext("2d");aa.translate(J,c);g.onmousemove=function(i){R=(i.clientX-J)/1e3;Q=(i.clientY+c)/1e3};while(X++=0){K=q*T-b*S;I=b*T+q*S;p=n*I-ac*U;r=ac*I+n*U;Z=l*r-ab*K;Y=ab*r+l*K;o=Z-C;m=Y-A;k=p+w;if(k>0){H=(64/k)>>0;aa.fillStyle="rgb("+(H-16)+","+(H*2-128)+","+(H+64)+")";j=W*30/k;L=j/2;aa.fillRect((o/k)*J-L,(m/k)*J-L,j,j)}}}},16);

This weekend Euskal took place and for quite a while I was considering doing a demo for it, but it wasn't until Friday night that I finally got free time for doing it (and the deadline was Saturday evening...).

Now that three.js was starting to get stable and also thanks to some sequencing code I had done some months ago I had no excuses to get working on it. So, from Friday midnight until Saturday afternoon, I managed to get this:



That's just a video of the demo, watch the actual demo.

Quite amazing how quickly you can get stuff done with javascript once you have the basic libs ready ;)

I need to check what's wrong with Opera, for some reason is not cleaning the screen. Apart from that it should work in all browsers (albeit quite slow in some, specially MacOS ones). Oh, I think it wouldn't work on Safari either, as it seems to be the only modern browser that doesn't support .ogg files.

In case anyone is interested, I've also shared the source code.

EDIT: Found the problem with Opera. Turns out context.clearRect() doesn't work if the context has been transformed with negative values. I'll let them fix their bug ;)

Here it's a project I made over the past weekend. I warn you, it's kinda weird. (Chrome 4+, Safari 5+, Firefox 4+ only)



You'll find some more iterations on my twitpic.

Now, lets give some context...

Nerd talk

Over the past few weeks I've seen people over twitter talking about this node.js thing. Because I've been quite into Javascript this year last week Jaakko was asking if I had played with node.js yet. I still didn't know what it was but he was kind enough to explain.

So node.js is like Apache but super optimised and lightweight, the coolest part is that you code your server in Javascript. Besides a small problem I had to deal with it was really easy to compile and install on my Ubuntu system.

Then there was this other hype thing, and that's websockets which basically is an open connection between the client and the server. You can send data to the server via websocket without having to send additional headers for each request.

So Jaakko persuaded me to try doing a node.js + websockets + harmony experiment. I was going to spend the weekend doing the WebGL renderer for three.js (which I did yesterday), and my mind was in another place, but somehow he convinced me ("fucking do it now" may have been the key ;D).

Ok, there are a bunch of node.js libraries to handle websockets. At first I was using socket.io but it seemed to use a lot of cpu. After a looking at the code a bit we find out that they JSON all the data that gets sent all the time. That can be useful but in this case it was quite a bit of overhead (and I think they're working to fix that). So I ended up using miksago's node-websocket-server which broadcasts the data to all the connected users as it got sent.

At this point is when you just need to sit down and get the thing working. Once it was working locally I got a slice in slicehost and in a matter of minutes the experiment was live. Considering that it costs $20 for 150gb of BW and that I received €60 from flattr last month I though it was worth spending that money on this project. (EDIT: At this rate I suspect I'll end up spending the full €60 on this...)

The first version of the code was pretty much brute force, every user was sending their X,Y position every time they moved their mouse. Yes, that's *a lot* of data per second. There has been 50gb of data transfered in just 4 days :P However, yesterday night I finally got a bit of time and improved a bit the data that gets transferred, and not only it consumes less bandwidth, the server's CPU is much happier (btw, thanks sole for the top tip :D).

Monitoring interactions

Leaving the technicalities aside, this experiment is very different from everything I've done until now. I've never seen how the people interact with my experiments. The feedback I get from them are emails, comments on the blog or tweets. Which basically reduces to a couple of sentences defining how the experience was. Which is good, but it's impossible to get a clear impression of the experience of the user.

With this one, I can see the people interacting. I can see people experimenting and writing scripts and doing unexpected things with the system (which were fun, but please, don't). I was even able to see people trying XSS exploits in realtime.

Considering how easy it was to put this together I won't be surprised if in a year or two, most of the websites use websockets to monitor how exactly their users navigate the site and improve the UX in realtime. And this is just an exmaple, imagine the possibilities!

Human behaviour

As expected, people like to draw male reproduction organs all over the place, and an annoying one was people scribbling on top of nice drawings. It was so annoying that I even had to tweak the brush algorithm to discourage such actions. The faster you move the cursor the less opacity the pencil has. Stupid solutions for stupid problems.

However, when humans want to collaborate the experiment becomes super fun. If you manage to get in a time that there isn't much people (rooms next?) and there is a collaborative person it can be great. For instance, the screenshot at the beginning of this post, I did some of the drawings on it in collaboration with a stranger, First I did a square with eyes and mouth, then he did another one on the site as if they were two squares that were about to collide and then he started doing the animal bits on the new one which I also followed by doing a joke to a classic internet image. It's also great when 4/5 users get organised and draw something together. 3-in-a-row games are also great :D

Last but not least, it's amazing to see, not only how one user interacts with the tool, but how they interact with the other people. That creates many games, they even create their own social rules for the place. And if that wasn't enough, I find it amazing to see how the cursors behave like "humans". If you put your cursor on top of another cursor, that other cursor is likely to get away. Like when someone you don't know get too close to you. The most interesting case was "Talentina", which I suspect is a little girl. She seemed happy and was drawing cute things like a giraffes and so on, but then someone will arrive and draw a penis on it. She would move the cursor away, like looking from the distance, and move away like in a slow/sad pace...

And lets not forget the fact that the history doesn't get saved, if you refresh you'll get a blank canvas. That's also a very strange behaviour to get used to, but people seem to be turning it into a feature :D

Unexpectedly fascinating.

EDIT: Running MacOS? Need a new screensaver? Cédric Raud has done something for you.

http://hxt.fr/vrac/Multiuser%20Sketchpad.screensaver.1.1.zip

Also... from "ops" in the comments, a recording of a session at 4000x :O

Some of you may have realised already that I've been using a Javascript FPS counter on some of my experiments.

Originally it had exactly the same look as the Actionscript version but, as far as I know, you can't query the system to get the amount of memory the application is using from Javascript. So 2 of the 4 common values of the Actionscript version were empty. That made me reconsider the design and tried to improve it a bit. And after using it for quite a while I think it's ready.



As you can see it has three modes: FPS, MS and MEM. Just click on the widget to switch between them. I may change MS to use greens instead of cyan to match the Actionscript version a bit more. Thoughts?

Unfortunately this time the code is not as simple as the Actionscript version, but I think it's still simple enough.

var stats = new Stats();
parentElement.appendChild( stats.domElement );

setInterval( function () {

    stats.update();

}, 1000 / 60 );

Find the source code, compressed file and more examples at the project page on github.

It's not the first time I mess with plane deformations, I also did some Pixel Bender experiments with them some years ago.

The principle of this technique is to apply a formula to every pixel of the output bitmap. If you know about shaders of Pixel Bender you know how this works.

Depending of the formula you're going to deform a source texture creating different forms. Javascript seems to be quite fast doing Math computations, the bottleneck, as usual, is drawing the pixels to the bitmap. So I thought it was worth porting the code and the formulas in order to get a base for it and have yet another experiment to test the new browser versions with.



Once again, using shaders we'll be able to do this kind of effects with much better resolution and speeds as soon as WebGL gets enabled on browsers by default.

Until then, feel free to take a look at the code and mess with the formulas.

Since I made three.js public I've been refactoring a bit some parts and experimenting a bit with potential performance improvement tricks. I won't go much into details, but maybe this snapshot will give you some idea of what I tried.



So that's nice and all that, but I was starting to get tired of flat colors so I decided it was time to try to implement textures and uv mapping. I took a look other engines to see how they were implementing these and the one that seemed the simplest and fastest was the one in pre3d which was done by Thatcher Ulrich. That also needed a bit of refactoring because I was storing the UV data on each triangle. Instead, it's better to store the UV on the geometry level so you can have different UV mappings (also called channels) for the same geometry.

Once I got that working it was time to do a bunch of demos and see how the browsers were able to handle it. Coincidently, Apple published their infamous HTML5 some days before.



From all the demos I got mostly interested on the VR one:



I know, I was also excited when I saw that message. Anyway, I managed to try the demos in Safari and this demo is pretty much a panorama. 6 planes textures and the camera inside. For that they used the CSS transforms which are only available in Safari at the moment. So I wondered if I could do that same effect using three.js and the new feature.

Turns out you don't need Safari to do a panorama. It's not as fast (not hardware accelerated), but hey, it works. I wasn't too happy about these textures and I looked for better ones for a proper demo. This is how it ended up:



That's better. Certainly slower as it's fullscreen but it has potential.

Next thing to try was using <video> as a texture for a plane. Adding a fake reflection on the floor for extra niceness.



Although it may seem much more advanced it's not. We just need to draw the <video> element to the texture applied in every frame.

Last, but not least interesting, was a Terrain Generator demo. The code to generate the height map and texture is not too different from the code I wrote some years ago while trying to do a raycaster. Only this time I needed to generate the texture instead and let the engine do the rendering part. After a couple of iterations [1, 2], this is the result:



I know, is so slow it's pointless, but I'm having fun doing all this anyway. I hope browsers will keep improving their implementation of <canvas> and maybe all this will have some potential. However, I have to say that I think a lot of things can be done with this. People have done amazing things with even more limitations.



Ah... so tempting... :P Anyway, I better start implementing a WebGL renderer to the engine.

As you may have seen, I've recently added a few Flattr buttons to some of my experiments.

Flattr? What? Here here:



I really hope it to succeed. It's like the old (and not so old) days, when the artists were funded by the people that enjoyed their work.

You may wonder, what was wrong with Paypal donations? Well... Paypal it's a bit like having to go to the bank every time you wanted to donate a coin.

Here it's my profile over there.