Add dimension to images

Platform-specific guides

The Augmented Images APIs in ARCore lets you build AR apps that can detect and augment 2D images in the user's environment, such as posters or product packaging.

You provide a set of reference images. ARCore uses a computer vision algorithm to extract features from the grayscale information in each image, and stores a representation of these features in one or more Augmented Image databases.

At runtime, ARCore searches for these features on flat surfaces in the user's environment. This lets ARCore detect these images in the world and estimate their position, orientation, and size if one is not provided.

Capabilities

ARCore can track up to 20 images simultaneously. ARCore will not simultaneously detect or track multiple instances of the same image.

Each Augmented Image database can store information up to 1,000 reference images. There's no limit to the number of databases, but only one database can be active at any given time.

Images can be added to an Augmented Image database at runtime, up to the 1,000-image per-database limit. It's possible to download previously created databases using a network connection.

When adding an image, it's possible to provide the physical size of the image to detect. Doing so will improve image detection performance.

• If no physical size is provided, ARCore estimates the size and refines this estimate over time.

• If a physical size is provided, ARCore uses the provided size and estimates the image's position and orientation, ignoring any discrepancy between apparent or actual size and the provided physical size.

ARCore can respond to and track images that are:

• Images that are fixed in place, such as a print hanging on a wall or a magazine on a table

• Moving images, such as an advertisement on a passing bus or an image on a flat object held by the user as they move their hands around.

Once ARCore begins tracking an image, it provides estimates for image position and orientation each frame. ARCore continuously refines these estimates as it gathers more data.

Once an image is detected, ARCore continues "tracking" the image's position and orientation, even when the image temporarily moves out of the camera view because the user has moved their device. When this happens, ARCore assumes that the image's position and orientation is static, and that the image is not itself moving through the environment.

All tracking happens on the device. No internet connection is required to detect and track images.

Requirements

Images must:

• Fill at least 25% of the camera frame to be initially detected.

• Be flat (for example, not wrinkled or wrapped around a bottle).

• Be in clear view of the camera. They should not be partially obscured, viewed at a highly oblique angle, or viewed when the camera is moving too fast due to motion blur.

CPU utilization and performance considerations

Depending on which ARCore features are already enabled, enabling Augmented Images might increase ARCore's CPU utilization. Consider disabling any unused features during times when your AR experience doesn't require them. This will make additional CPU cycles available to your app, and improve thermal performance and battery life.

For additional information, see performance considerations.

Best practices

Tips for selecting reference images

• The image's resolution should be at least 300 x 300 pixels. Using images with high resolution does not improve performance.
• Reference images can be provided in PNG or JPEG file format.
• Color information is not used. Both color and equivalent grayscale images can be used as reference images or by users at runtime.
• Avoid images with heavy compression as this interferes with feature extraction.
• Avoid images with that contain a large number of geometric features, or very few features (e.g. barcodes, QR codes, logos and other line art) as this will result in poor detection and tracking performance.
• Avoid images with repeating patterns as this also can causes issues with detection and tracking.

• Use the arcoreimg tool included in the ARCore SDK to get a quality score between 0 and 100 for each image. We recommend a quality score of at least 75. Here are two examples:

  Example image 1	Example image 2
  Score: 0	Score: 100
  contains repetitive geometric features	sufficient resolution; contains many unique feature

Tips for creating the image database

• Use the arcoreimg tool for Android to generate an image database file. This tool is available for only Android and Android NDK development. It is built into the Unity SDK and ARCore Unreal plugin.
• The database stores a compressed representation of the features extracted from the grayscale data in the reference images. Each image entry occupies about 6 KB.
• It takes about 30 ms to add an image to the database at runtime.
  • Add images on a worker thread to avoid blocking the UI thread.
  • Or, if possible, add images at compile time using the arcoreimg tool included in the ARCore SDK.
• If you know the expected physical size of an image, specify it. This information improves detection and tracking performance, especially for large physical images (over 75 cm).
• Avoid keeping many unused images in the database because there's a slight impact on system performance due to increased CPU utilization.

Tips for optimizing tracking

• If your image will never move from its initial position (for example, a poster affixed to a wall), you can attach an anchor to the image to increase tracking stability.
• The physical image must occupy at least 25% of the camera image for initial detection. You can prompt users to fit the physical image in their camera frame.
• Do not use the image's pose and size estimates until the image's tracking state is full tracking. When an image is initially detected by ARCore, and no expected physical size was specified, its tracking state will be paused. This means that ARCore has recognized the image, but has not gathered enough data to estimate its location in 3D space.