The Augmented Images APIs in ARCore lets you build AR apps that can detect and augment 2D images in the user's environment, such as posters or product packaging.
You provide a set of reference images. ARCore uses a computer vision algorithm to extract features from the grayscale information in each image, and stores a representation of these features in one or more Augmented Image databases.
At runtime, ARCore searches for these features on flat surfaces in the user's environment. This lets ARCore detect these images in the world and estimate their position, orientation, and size if one is not provided.
ARCore can track up to 20 images simultaneously. ARCore will not simultaneously detect or track multiple instances of the same image.
Each Augmented Image database can store information up to 1,000 reference images. There's no limit to the number of databases, but only one database can be active at any given time.
Images can be added to an Augmented Image database at runtime, up to the 1,000-image per-database limit. It's possible to download previously created databases using a network connection. See the developer guide for details.
When adding an image, it's possible to provide the physical size of the image to detect. Doing so will improve image detection performance.
If no physical size is provided, ARCore estimates the size and refines this estimate over time.
If a physical size is provided, ARCore uses the provided size and estimates the image's position and orientation, ignoring any discrepancy between apparent or actual size and the provided physical size.
ARCore can respond to and track images that are:
Images that are fixed in place, such as a print hanging on a wall or a magazine on a table
Moving images, such as an advertisement on a passing bus or an image on a flat object held by the user as they move their hands around.
Once ARCore begins tracking an image, it provides estimates for image position and orientation each frame. ARCore continuously refines these estimates as it gathers more data.
Once an image is detected, ARCore continues "tracking" the image's position and orientation, even when the image temporarily moves out of the camera view because the user has moved their device. In doing so, ARCore assumes that the image's position and orientation is static, and that the image is not itself moving through the environment.
tracking method to determine
whether ARCore is currently tracking an image based on where the image is
currently detected in the camera view (
"full tracking"), or based on its
most recently detected position in the world (
"last known pose").
All tracking happens on the device. No internet connection is required to detect and track images.
Fill at least 25% of the camera frame to be initially detected.
Be flat (for example, not wrinkled or wrapped around a bottle).
Be in clear view of the camera. They should not be partially obscured, viewed at a highly oblique angle, or viewed when the camera is moving too fast due to motion blur.
CPU utilization and performance considerations
Depending on which ARCore features are already enabled, enabling Augmented Images might increase ARCore's CPU utilization. Consider disabling any unused features during times when your AR experience doesn't require them. This will make additional CPU cycles available to your app, and improve thermal performance and battery life.
For additional information, see performance considerations.
Tips for selecting reference images
- The image's resolution should be at least 300 x 300 pixels. Using images with high resolution does not improve performance.
- Reference images can be provided in PNG or JPEG file format.
- Color information is not used. Both color and equivalent grayscale images can be used as reference images or by users at runtime.
- Avoid images with heavy compression as this interferes with feature extraction.
- Avoid images with that contain a large number of geometric features, or very few features (e.g. barcodes, QR codes, logos and other line art) as this will result in poor detection and tracking performance.
- Avoid images with repeating patterns as this also can causes issues with detection and tracking.
arcoreimgtool included in the ARCore SDK to get a quality score between
100for each image. We recommend a quality score of at least 75. Here are two examples:
Example image 1 Example image 2 Score:
contains repetitive geometric features sufficient resolution; contains many unique feature
Tips for creating the image database
- The database stores a compressed representation of the features extracted from the grayscale data in the reference images. Each image entry occupies about 6 KB.
- It takes about 30 ms to add an image to the database at runtime.
- Add images on a worker thread to avoid blocking the UI thread.
- Or, if possible, add images at compile time using the
arcoreimgtool included in the ARCore SDK.
- If you know the expected physical size of an image, specify it. This information improves detection and tracking performance, especially for large physical images (over 75 cm).
- Avoid keeping many unused images in the database because there's a slight impact on system performance due to increased CPU utilization.
Tips for optimizing tracking
- If your image will never move from its initial position (for example, a poster affixed to a wall), you can attach a global anchor to the image to increase tracking stability.
- The physical image must occupy at least 25% of the camera image for
initial detection. You can prompt users to fit the physical image in their
camera frame with the
FitToScan.pngasset. See the developer guide for an example of this prompt.
- Do not use the image's pose and size estimates until the image's tracking
"full tracking". When an image is initially detected by ARCore, and no expected physical size was specified, its tracking state will be
"paused". This means that ARCore has recognized the image, but has not gathered enough data to estimate its location in 3D space.