Inside FLARManager: Basic Augmented Reality

<– 2D Marker Tracking | Loading Collada Models –>
So, you want to create augmented reality in a browser. You saw the GE ad, you tried stuffing your cat into a USPS box, and now you want a piece.

This tutorial will run you through the basics of getting an augmented reality application up and running. You might want to try the 2D tutorial first, to get a handle on a Papervision-less FLARManager application. Or, you might just want to dive right in.


FLARManager Tutorial: 3D and Augmented Reality

Download source for this tutorial here:
and place it in the root of the /src/ FLARManager folder.
Print out any of the pattern .pngs in /resources/flarToolkit/patterns, found in the FLARManager distro, to use with the tutorial.
This tutorial will demonstrate how to draw a cube wherever the tracker sees a marker. We’ll start with just one cube for now; things get more complex when you want to support multiple cubes tied to multiple markers.

In the application class’ constructor, we wait for the class to be added to the stage, so that we have a reference to the stage:
this.addEventListener(Event.ADDED_TO_STAGE, this.onAdded);

Once the application is added to the stage, we can begin setting things up. First, we create a FLARManager instance and pass it the path to an external xml configuration file. For now, we’ll use flarConfig.xml (located in the /resources/flar/ folder in the FLARManager distro). We also pass an instance of FLARToolkitManager, to use the FLARToolkit tracking library, and a reference to the stage.
var flarManager:FLARManager = new FLARManager("flarConfig.xml", new FLARToolkitManager(), this.stage);

Once you’re comfortable with the basics of FLARManager, you can edit the config file, or create your own, as you see fit. More information on configuration options lives here.

We want to see the video capture, so let’s add it to the stage. FLARManager creates a default video source all ready to go, though the source can also be modified via the configuration file.

FLARManager uses an event model to notify any interested parties of newly-detected markers, changes in already-detected markers, and marker removal. We add FLARMarkerEvent handlers to respond to these changes.

this.flarManager.addEventListener(FLARMarkerEvent.MARKER_ADDED, this.onMarkerAdded);
this.flarManager.addEventListener(FLARMarkerEvent.MARKER_UPDATED, this.onMarkerUpdated);
this.flarManager.addEventListener(FLARMarkerEvent.MARKER_REMOVED, this.onMarkerRemoved);

We’ll write our event handlers in just a a bit. First, we wait for FLARManager to initialize before setting up the Papervision3D environment.

this.flarManager.addEventListener(Event.INIT, this.onFlarManagerInited);


Setting up Papervision3D

Once FLARManager has finished initializing, we can set up the Papervision3D environment:

private function onFlarManagerInited (evt:Event) :void {
    this.flarManager.removeEventListener(Event.INIT, this.onFlarManagerInited);
    this.scene3D = new Scene3D();
    this.viewport3D = new Viewport3D(this.stage.stageWidth, this.stage.stageHeight);
    this.camera3D = new FLARCamera_PV3D(this.flarManager, new Rectangle(0, 0, this.stage.stageWidth, this.stage.stageHeight));
    this.renderEngine = new LazyRenderEngine(this.scene3D, this.camera3D, this.viewport3D);
    this.pointLight3D = new PointLight3D();
    this.pointLight3D.x = 1000;
    this.pointLight3D.y = 1000;
    this.pointLight3D.z = -1000;
    this.addEventListener(Event.ENTER_FRAME, this.onEnterFrame);

Papervision has to re-render the scene every frame, so we add an ENTER_FRAME event handler in which we’ll do that. (We’ll get to that part in a sec.)


Tracker camera parameters

This section is extra credit, but thought you might want to know about that FLARCamera_PV3D line…

Many tracking libraries compensate for distortion caused by the camera lens by referring to an external camera parameters file. With flare*tracker and flare*NFT, this file is /resources/flare/cam.ini; for FLARToolkit it is /resources/flarToolkit/FLARCameraParams.dat. FLARManager has a camera package that contains camera classes that can parse and apply the data within these files to incorporate this compensation into the displays generated by different 3D frameworks.

FLARCamera_PV3D requires the information from the camera parameters file, but FLARManager cannot provide this until it has loaded and parsed this file. Therefore, we wait to initialize the Papervision3D environment until after FLARManager has initialized. Once there, we pass a reference to FLARManager into FLARCamera_PV3D, from where the loaded camera parameters are extracted and applied.


Create the Cube

Let’s set up a single Cube instance that we’ll map to the detected marker:

    var cubeMaterial:FlatShadeMaterial = new FlatShadeMaterial(this.pointLight3D, 0xFF1919, 0x730000);
    var materialsList:MaterialsList = new MaterialsList({all: cubeMaterial});
    var cube:Cube = new Cube(materialsList, 40, 40, 40);
    cube.z += 20;
    // create a container for the cube, that will accept matrix transformations.
    this.cubeContainer = new DisplayObject3D();

We place the Cube inside of a DisplayObject3D so that we can lift the Cube 20 pixels (in the z-axis) above the center of the marker, so it appears to be sitting on the marker instead of embedded in it. If we moved the Cube up 20 pixels and then applied the transformation matrix directly to it, the 20px z-coordinate would get overwritten, and the Cube would appear stuck in the middle of the marker.


Responding to FLARMarkerEvents

Now that FLARManager and Papervision3D are both set up, we can make the two work together by drawing Papervision3D objects when handling FLARMarkerEvents coming from FLARManager.

private function onMarkerAdded (evt:FLARMarkerEvent) :void {
    this.cubeContainer.visible = true;
    this.activeMarker = evt.marker;
private function onMarkerRemoved (evt:FLARMarkerEvent) :void {
    this.cubeContainer.visible = false;
    this.activeMarker = null;

For the purposes of this tutorial, we’re simply toggling the visibility of the Cube as a marker is added and removed from the camera’s view. We’re also keeping track of the active FLARMarker, from which we’ll extract and apply the transformation matrix that makes the Cube appear to be tethered to the marker.

For more on FLARMarkerEvents, see the 2D tutorial.


Updating the Cube

As mentioned earlier, Papervision has to re-render the scene every frame. We’ll take advantage of this opportunity to apply the latest transformation matrix to the Cube, to make it look like it’s Augmenting our Reality.

private function onEnterFrame (evt:Event) :void {
    this.cube.transform = PVGeomUtils.convertMatrixToPVMatrix(this.activeMarker.transformMatrix, this.flarManager.flarSource.mirrored);

Note that the FLARMarker instance we’re tracking as this.activeMarker will be updating itself continuously; FLARManager handles this for us. We could listen for changes as MARKER_UPDATED events, but since we have to render the Papervision scene every frame, we can just grab the updated information from this.activeMarker while we’re at it.

You might also notice that long, ugly method call, PVGeomUtils.convertMatrixToPVMatrix. (Unfortunately at times like this, I’m a firm believer in self-commenting code.) flare*tracker, FLARToolkit, and other tracking libraries use different coordinate systems than Papervision (and Flash 3D, and Away3D, and Sandy, and Alternativa), so we have to convert the tracking library’s transformation matrix into numbers that make sense to Papervision before applying the matrix to the Cube. Coincidentally, PVGeomUtils has just the method for us!

That should take care of it — you should now see a colored cube perched gently on the marker in your hand!

To handle multiple cubes and multiple markers, we have to be smarter about managing all the DisplayObject3D instances and active FLARMarker instances. One way to do this is laid out in this example:

Also, FLARManager supports arbitrary aspect ratios. No need to stick with 4:3! Here’s a 16:9 example, if you want to get all cinematic:
<– 2D Marker Tracking | Loading Collada Models –>