Spatial Augmented Reality Gives a Different View on Reality

A few months ago, a little show-reel by made the rounds on the net. It showed that new media doesn’t have to be on the Internet or even on your screen. They use buildings as canvasses. Buildings seem to come to life as bricks move around on the wall and strange creatures lurk behind windows. It’s a young artform, known under several names, like “projection mapping”, “geometric projection” or most often “spatial augmented reality”. Before I start experimenting with it, this post is an overview of what information I could find on the net.

If you missed the movie, here are 7 minutes of inspiration (via Digital Urban):

Basically what happens is they put one bad-ass projector in front of a building and calibrate the software, so that it knows where all the buildings features are. Once the building is entered into a 3D package, they can add artificial elements to the projection.

Now this might sound easy, but trust me, it is not. Reconstructing the building in 3D and exactly positioning the projector to be aligned with the computer image is only the start. Next you need images that look good from different sites. Because there’s no “real” 3D here, it is easy to get results that only look good from one particular angle.

On the Internet, this spatial augmented reality seems to be rather unknown. There are many scientific papers if you search google, but there aren’t too many practical tutorials. Here are the 5 most interesting links I found, that should get you up to speed quickly:

  1. Spatial Augmented Reality is a free downloadable book describing many types of augmented reality. It goes into great depth on the existing techniques and it gives an overview of the necesarry Maths. The material is not easy, but the explanation is very clear. Over 300 pages of knowledge will get you quickly started. It’s certainly worth buying the paper version to have it always available next to your computer.
  2. An impressive experiment in finding objects automatically. It shows you how to digitize the environment based on projecting known patterns (and you’ll need sensors on the object, which might not always be feasible). It’s a bit low on technical details, but if you want more information, you can always check out Johnny Chung Lee’s PhD thesis, on which this is based.
  3. This article is pretty old, but it is still a very nice tutorial on mapping the real world on a virtual one. Once this 3D world is created you can enhance it and project it back on the real one.
  4. This final blog post in the list shows a real life demo. It documents some of the hardware setup and used 3D objects (also check the links), but sadly no video of the actual result.
  5. vvvv is the only toolkit I could find that is actually aimed at applications like this (and many other VJ’ing applications). They have an in-depth tutorial on how to set things up and what to watch out for.

Cool stuff! My next step is obtaining a decent projector for an affordable price. To be continued …