augmenting

Layar announced last week their new 3D feature coming in November. This feature will enable 3D rendering for Layar’s API, so that you can have something like this:

on the mobile application developed using Layar.

Interesting because: while we’ve seen a lot of AR apps on the desktop using 3D renderings (from Total Immersions to all the projects using ARToolKits, FLAR.. etc.) the AR apps on mobile have all been limited to drawing flat sprites/billboards on top of live video. Also note that this is *not* using a marker but location data in order to draw the 3D objects. Could be interesting to learn how they determine camera translation and smooth out the jitter.

However, 3D in AR on mobile phone still has lots of ways to go. Layar 3D: “With the limited processing power of phones we try to keep 3D objects below 1000 polygons. The platform can process objects up to 5000 polygons but starts to slow down.” Which means we probably won’t be able to get a lot of smooth-looking stuff w/ the current smart phones, but alas, it is a step forward.

Also, this looks like something out of Fringe:

Source: http://layar.com/3d/

We spent some good number of hours last week looking into analyzing the video feed on the iPhone in order to do marker tracking for AR. The common knowledge amongst the developers is that it is currently not supported by Apple, but we wanted to look into what exactly needed to happen and if there is a way to get around it using a combination of techniques available.

Well, in short, the answer’s a big disappointing no. At least not just yet with the SDK. There is no quick way to get to the video stream in order to analyze the frame. While image capturing provides enough data to do image analysis on a single frame, drawing on top of the image then re-analysing the next frame, is, however, impossible because any elements that were drawn onto to the video will get captured as well. In addition, the frame rate would be too low to make the application seem realistic (1 fps at best).

Our assumption is that marker-based tracking iPhone apps that might be available to developers out there (as we’ve seen with ARToolKit 4.4 demo video) must be using non-SDK code, aka jailbroken method.

AR using only location based data, however, is available in the SDK and ready to roll.  iPhone 3.1 SDK made available the ability to draw sprites on top of video, and we’ve seen way-finding applications already available in the app store.  (see now-available apps in the previous post)

Also, a techy read:  Why 3d markerless tracking is difficult for mobile augmented reality

AR apps using location data on the iPhone (no video analyzing). Seems like these are available now on the app store (as of 21/09/09):

New York Nearest Subway from acrossair – NYC’s subway map

London Nearest Tube, also from acrossair – London’s Tube map

Discover Anywhere Transit – Transit information for several cities in North America.

While certainly cutting edge and putting new technology into practical use, the actual usefulness of these augmented maps remain to be seen, however.

These effects are done in post-production, but would be super sweet as mobile AR:

street tests from Najork on Vimeo.

Over the past week, we’ve looked into several “tech tools” for doing augmented reality applications. Here’s a round up of what we’ve found so far:

1. Marker-based AR

The one and only open-source tool available seems to be the ARToolKit library, of which many variation and flavors have been created using the GNU-licensed library to make AR development faster and in more areas, some of these include:

OSGART:  ARToolKit + OpenSceneGraph

OSGART is a library that simplifies the development of Augmented Reality or Mixed Reality applications by combining the well-known ARToolKit tracking library with OpenSceneGraph.

With OSGART, users gain the benefit of all the features of OpenSceneGraph (high quality renderer, multiple file type loaders, community nodekits like osgAL, etc.) directly in their augmented reality (AR), mixed reality (MR) or mediated reality applications.

FLARToolkit
An AS3 library for developing AR in Flash.

FLARManager

FLARManager is a lightweight framework that makes building FLARToolkit augmented reality applications easier. it is compatible with a variety of 3D engines (currently: Alternativa3D,Away3D, Papervision3D, and Sandy3D), and provides a more robust event-based system for managing marker addition, update, and removal. it supports detection and management of multiple patterns, and multiple markers of a given pattern.

simpleARToolKit

a very primitive and limited Processing library …made as a wrapper from another open source project jARToolKit which is a Java port of the original ARToolKit. It only works with an OpenGL sketch in Windows.

BuildAR
a “no-programming required” software tool to generate your own AR scene on desktop. From HitLab, developers of ARToolKit. Currently (Sept 09) is PC only.

ARToolWorks
Distributer of commercial license for ARToolKit

Our notes: we’ve experiment with using ARToolKit with Cocoa (our bread n’ butter) – not as easy as it sounds, but eventually got it to work.  The difficulty for all the AR development seems to be the 3D drawing part.  With just ARToolKit, you’d have to do the drawing in OpenGL yourself – which, perhaps, explains the existence of OSGART.  The Flash route probably gives the developer more flexibility with 3D drawing, although we haven’t tried it ourselves.

2.  Mobile AR

See earlier post on Locative AR, these are markerless AR using the smartphone’s data.

There are lots of development done for the Android phone already, but really the big game player is the iPhone (and is the one we’re most interested in).  Most of the dev for AR on the iPhone is limited to the developer themselves at this point (as of Sept 09), with a few apps that slipped through to the app store such as Yelp! Monocle and the Paris Subway Map.  Both are using the combination of GPS, magnetometer and accelerometer data, then overlaying the info graphic on the video feed.  Useful, maybe.  Suspect that the novelty might wear off after a while if doesn’t offer any more functionality, but all is still in development.

Acrossair claims they have iPhone AR apps ready and waiting for launch when 3.1 releases (which happened yesterday (10 Sept 09), but we’re not sure if Apple released the video API w/ 3.1).

On the Android:  Wikitude is doing lots of ground work for this type of stuff.  Layar as well.

Tools:  Wikitude API for Android, Layar API for Android

ARToolKit has a version of the library that works with iPhone, but not released yet.

3.  Marker-less AR

This is where augmented reality gets more exciting and a bit out of “geek-mode”.  The basis is the same, but using advanced camera vision technique, the software would be able to do image recognition and, without using the black-and-white markers, render the computer generated visuals on top of it.  Development and applications in this area are happening, but not as wide spread as the opensource marker-based (which is thanks to ARToolKit and others).  We assume that in the future, AR markers will be obsolete in a few years’ time.

Proprietary tools:
Total Immersion – lots and lots of commercial and advertising work for big companies from consumer products to expos, games, and film industries.  Impressive portfolio.  They also offer a packaged software.  PC-based and final results require DirectX plugin on the user part.

Metaio – offers PC-based software for developing AR applications.  Also PC based.

Code libaries:
As mentioned above, augmented reality is based on two things working together:  camera vision to determine where/what the user is looking at and rending 3D onto that video frame in real time.  This combination could be technically possible using OpenCV and OpenGL, both libraries packages are free and open for commercial usage.

Overall Note: camera vision requires a lot of resources, so while it is possible to do complicated AR scenes on today’s desktop computers, we’re wondering a bit about the capabilities on the smartphones, especially for the marker-less option.

Great use of AR for architectural presentations:

via arvertising

Video explanation of the technology behind locative AR on mobile from GigantiCo blog:

Our path to the world of augmented reality probably began like most curious coders out there:  it started with a link to some YouTube video and a “hey that’s cool, let me figure out how they did that” and naturally this led to Googling, finding some opensource codes, staying up late and opening up 20 more browser windows, etc.

Wait, actually, maybe it was seeing that GE ad… or was it that Japanese video… or was it this blog post?

Anyway, there’s a lot out there, and it’s steadily gaining street creds despite the gimmicky nature of 3D graphics.  Now that the venerable iPhone is about to get some AR applications in the App Store (they said September, right?), the time seems right to pay attention to this thing.  We’re starting this website as a part of our R&D effort to collect resources, notes, and track the world of augmented reality development, mobile platforms and otherwise.  Who knows, it might be useful.