Augmented reality using a webcam and Flash
Augmented reality (AR) is a field of computer science that involves combining the physical world and an interactive, three-dimensional virtual world. This article takes a look at augmented reality, its current uses, and its future potential. Then you'll dig in and see how to apply this exciting technology using Adobe Flash CS4 Professional, Adobe Flash Player 10, and a webcam.
While mainstream audiences are now becoming aware of AR, it is not new. Its background is intertwined with decades of computer science development. Virtual reality (VR), AR's more familiar counterpart, is the replacement of the user's physical reality (particularly that which is experienced through sight and hearing) with a computer-generated reality. The idea of a virtual experience is exciting—creating entertaining and educational sensory encounters that do not exist in our everyday lives.
From a consumer standpoint, it seems that AR advances have come out of nowhere to surpass VR advances. The acceleration in AR technology is due to two major factors: First, users are still experiencing reality, so believability is easier to achieve. Adding simple graphics (such as text or simple shapes) and color effects (such as night vision or thermal vision) to reality creates a better user experience. The user is still seeing a mostly familiar world. Second, this more subtle use of computer graphics is less expensive with today's technology, making it more feasible than VR.
The video game industry has released major augmented reality products for more than a decade. The EyeToy for Sony PlayStation 2 and 3 takes input from an inexpensive video camera and composites the live video feed with CG onto the TV screen. This toy detects the user's body position in front of the camera as an alternative input method to the typical joystick or gamepad, deepening the user's immersion into the game world. Detection schemes continue to advance, allowing for more engaging interaction.
There are AR applications outside of console games, including military and consumer products, too. Night-vision goggles and targeting-assistance technology help marksmen in battle, and children's stories come to life with AR-enhanced books. The uses are vast.
With AR-enhanced books, each page of a real-world book is outfitted with a marker—a pattern that is detectable by AR. As the child turns each page, a computer is able to place a virtual 3D animation on top of a physical image printed on paper. In this article, you will learn to create this type of AR experience. Let's dig in!
Let's create a project in which a webcam will capture the user, holding a marker image in an arbitrary position, and the application will augment the webcam feed with a 3D model overlaid onto the marker's position in a believable way. You can get this project up and running in less than an hour.
With Flash Player 10, developers have the most robust toolset yet for rich application development. Flash Player handles the heavy lifting such as video input, pixel-level image manipulation, and the heavy number crunching afforded with ActionScript 3. All you need is a webcam, a few open source libraries, and Flash CS4 to get the job done.
Before coding a Flash CS4 project, you need to prepare the libraries and assets. Once the project is set up, most changes will happen within the document class (see Figure 1). To get started:
1. Create a new project in Flash CS4 by choosing File > New > Flash File (ActionScript 3.0), and save the project asAugmentedReality.fla inside an AugmentedReality folder on your desktop.
2. Create a document class by choosing File > New > ActionScript File, and save it as AugmentedReality.as within the AugmentedReality folder.
3. Set this file as the project's document class to the AugmentedReality class file by choosing Window > Property > Class. The class file will be empty at this point.
4. Create a libs folder inside the AugmentedReality folder, and add it to the project's library path by choosing File > Publish Settings > Flash > ActionScript 3.0 > Library Path > [Folder Icon].
5. Download the three libraries discussed in the next section, and add each SWC to the libs folder.
6. Create an Assets folder and add the 3D Collada model file and two AR data files.
Figure 1. Augmented reality project folder after setup
A great benefit of ActionScript3's object-oriented programming model is that developers can importexisting libraries of code to save ramp-up time and vastly expand thedevelopment power of Flash. This implementation of AR takes advantageof three such libraries, each with a specific role in the project:
· Flex SDK code library: Processes the Embed metatag needed to import FLAR data files.
· Spark Project's Flash Augmented Reality code library (FLARToolkit): Handles marker graphic detection.
· Papervision3D code library (PV3D): Handles the importing, positioning, and rendering of the 3D model.
TheFlash Augmented Reality Toolkit (FLARToolkit) library holds much of themagic for this application. It will introspect the webcam image for aparticular graphic image and judge where in real space to map the 3Dmodel. The FLARCameraParams.dat file, which comes with the toolkit,will be used as-is. It corrects webcam distortion and helps the toolkitwith its general detection routine. The graphic marker imageinformation is held within the FLARPattern.pat file. The file is madeto work with a particular marker graphic image (see Figure 2). If youwant to use a different graphic, create a JPEG image with your favoritedrawing program, and use the FLAR Marker Generator(AIR, 322K) to generate a new pattern file.
Figure 2. Tower marker graphic
The marker image is a graphic drawn, printed, and shown to the endapplication as it runs. FLAR, with help from the marker data file andparameters file, will detect this shape via webcam. To design agraphic, fit the desired shape within a white square that is centeredwithin a larger black square. Keep your shape simple for best results.
Notethe sun shape in the graphic. This helps the application detect whichway is up for the marker and assists its angle and rotation detection.Your design doesn't need to have this shape per se, but it should beasymmetrical in some way for best results.
Also, you'll needa 3D model in the Collada format that Papervision3D requires. Colladamodels can be created and exported using many popular 3D modelingtools, including the open source Blender3D.
TheAugmentedReality.as document class is where all of the project's codelives. The following code shows the completed class. As you can see,much of the complexity of FLAR and Papervision3D for the project ishidden within the imported libraries and is therefore out of the way:
package {
//--------------------------------------
// Imports
import flash.display.BitmapData;
import flash.display.Sprite;
import flash.events.Event;
import flash.media.Camera;
import flash.media.Video;
import flash.utils.ByteArray;
import org.libspark.flartoolkit.core.FLARCode;
import org.libspark.flartoolkit.core.param.FLARParam;
import org.libspark.flartoolkit.core.raster.rgb.FLARRgbRaster_BitmapData;
import org.libspark.flartoolkit.core.transmat.FLARTransMatResult;
import org.libspark.flartoolkit.detector.FLARSingleMarkerDetector;
import org.libspark.flartoolkit.pv3d.FLARBaseNode;
import org.libspark.flartoolkit.pv3d.FLARCamera3D;
import org.papervision3d.lights.PointLight3D;
import org.papervision3d.materials.shadematerials.FlatShadeMaterial;
import org.papervision3d.materials.utils.MaterialsList;
import org.papervision3d.objects.parsers.DAE;
import org.papervision3d.objects.primitives.Cube;
import org.papervision3d.render.BasicRenderEngine;
import org.papervision3d.scenes.Scene3D;
import org.papervision3d.view.Viewport3D;
// Class Definition
public class AugmentedReality extends Sprite
{
// Class Properties
// 1. WebCam
private var video : Video;
private var webcam : Camera;
// 2. FLAR Marker Detection
private var flarBaseNode : FLARBaseNode;
private var flarParam : FLARParam;
private var flarCode : FLARCode;
private var flarRgbRaster_BitmapData : FLARRgbRaster_BitmapData;
private var flarSingleMarkerDetector : FLARSingleMarkerDetector;
private var flarCamera3D : FLARCamera3D;
private var flarTransMatResult : FLARTransMatResult;
private var bitmapData : BitmapData;
private var FLAR_CODE_SIZE : uint = 16;
private var MARKER_WIDTH : uint = 80;
[Embed(source="./assets/FLAR/FLARPattern.pat", mimeType="application/octet-stream")]
private var Pattern : Class;
[Embed(source="./assets/FLAR/FLARCameraParameters.dat", mimeType="application/octet-stream")]
private var Params : Class;
// 3. PaperVision3D
private var basicRenderEngine : BasicRenderEngine;
private var viewport3D : Viewport3D;
private var scene3D : Scene3D;
private var collada3DModel : DAE;
// Fun, Editable Properties
private var VIDEO_WIDTH : Number = 640; //Set 100 to 1000 to set width of screen
private var VIDEO_HEIGHT : Number = 480; //Set 100 to 1000 to set height of screen
private var WEB_CAMERA_WIDTH : Number = VIDEO_WIDTH/2; //Smaller than video runs faster
private var WEB_CAMERA_HEIGHT : Number = VIDEO_HEIGHT/2; //Smaller than video runs faster
private var VIDEO_FRAME_RATE : Number = 30; //Set 5 to 30. Higher values = smoother video
private var DETECTION_THRESHOLD : uint = 80; //Set 50 to 100. Set to detect marker more accurately.
private var DETECTION_CONFIDENCE : Number = 0.5; //Set 0.1 to 1. Set to detect marker more accurately.
private var MODEL_SCALE : Number = 0.8; //Set 0.1 to 5. Set higher to enlarge model
// Fun, Editable Properties: Load a Different Model
private var COLLADA_3D_MODEL : String = "./assets/models/licensed/hummer/models/hummer.dae";
// Constructor
/**
* The constructor is the ideal place
* for project setup since it only runs once.
* Prepare A,B, & C before repeatedly running D.
**/
public function AugmentedReality()
//Prepare
prepareWebCam(); //Step A
prepareMarkerDetection(); //Step B
preparePaperVision3D(); //Step C
// Repeatedly call the loop method
// to detect and adjust the 3D model.
addEventListener(Event.ENTER_FRAME, loopToDetectMarkerAndUpdate3D); //Step D
}
// Methods
* A. Access the user's webcam, wire it
* to a video object, and display the
* video onscreen.
private function prepareWebCam() : void
video = new Video(VIDEO_WIDTH, VIDEO_HEIGHT);
webcam = Camera.getCamera();
webcam.setMode(WEB_CAMERA_WIDTH, WEB_CAMERA_HEIGHT, VIDEO_FRAME_RATE);
video.attachCamera(webcam);
addChild(video);
...
kaemo