Augmented Images Developer Guide for Android NDK

Learn how to use Augmented Images in your own apps.

Prerequisites

Make sure that you understand fundamental AR concepts and how to configure an ARCore session before proceeding.

Create an image database

Create an ArAugmentedImageDatabase to store reference images. There are two ways:

  • Create an empty database
ArAugmentedImageDatabase* ar_augmented_image_database = NULL;
ArAugmentedImageDatabase_create(ar_session, &ar_augmented_image_database);
 
std::string database_buffer;
util::LoadFileFromAssetManager(asset_manager_, "sample_database.imgdb",
                               &database_buffer);
uint8_t* raw_buffer = reinterpret_cast<uint8_t*>(&database_buffer.front());

ArAugmentedImageDatabase* ar_augmented_image_database = NULL;
const ArStatus status = ArAugmentedImageDatabase_deserialize(
    ar_session, raw_buffer, database_buffer.size(),
    &ar_augmented_image_database);

Database files can be created either with the Image Tool or by calling ArAugmentedImageDatabase_serialize().

Add images to database

This step is optional if your desired reference images have already been loaded from the database file. To add an image at runtime, call ArAugmentedImageDatabase_addImage() as shown below. Refer to the augmented_image_c sample app for functions in the util namespace.

int32_t width, height, stride, index;
uint8_t* image_pixel_buffer = nullptr;
constexpr const char kSampleImageName[] = "default.jpg";
bool load_image_result = util::LoadImageFromAssetManager(
    kSampleImageName, &width, &height, &stride, &image_pixel_buffer);

uint8_t* grayscale_buffer = nullptr;
util::ConvertRgbaToGrayscale(image_pixel_buffer, width, height, stride,
                             &grayscale_buffer);

int32_t grayscale_stride = stride / 4;
const ArStatus status = ArAugmentedImageDatabase_addImage(
    ar_session_, ar_augmented_image_database, kSampleImageName,
    grayscale_buffer, width, height, grayscale_stride, &index);

// If the physical size of the image is known, you can instead use
//     ArStatus ArAugmentedImageDatabase_addImageWithPhysicalSize
// This will improve the initial detection speed. ARCore will still actively
// estimate the physical size of the image as it is viewed from multiple
// viewpoints.

delete[] image_pixel_buffer;
delete[] grayscale_buffer;

The index and name values are used later to identify which reference image was detected.

Enable image tracking

Configure your ARCore session to track images by registering the image database:

ArConfig_setAugmentedImageDatabase(ar_session_, ar_config,
                                   ar_augmented_image_database);
const ArStatus status = ArSession_configure(ar_session_, ar_config);

During the session, ARCore uses feature points from the camera image to match against those in the image database.

Find augmented images in an AR session

To get the matched images, poll for updated ArAugmentedImages in your frame update loop.

// Update loop, in onDrawFrame
ArTrackableList* updated_image_list = nullptr;
ArTrackableList_create(ar_session_, &updated_image_list);
ArFrame_getUpdatedTrackables(
    ar_session_, ar_frame_, AR_TRACKABLE_AUGMENTED_IMAGE, updated_image_list);

int32_t image_list_size;
ArTrackableList_getSize(ar_session_, updated_image_list, &image_list_size);

for (int i = 0; i < image_list_size; ++i) {
  ArTrackable* ar_trackable = nullptr;
  ArTrackableList_acquireItem(ar_session_, updated_image_list, i,
                              &ar_trackable);
  ArAugmentedImage* image = ArAsAugmentedImage(ar_trackable);

  ArTrackingState tracking_state;
  ArTrackable_getTrackingState(ar_session_, ar_trackable, &tracking_state);

  int image_index;
  ArAugmentedImage_getIndex(ar_session_, image, &image_index);

  if (tracking_state == AR_TRACKING_STATE_TRACKING) {
    util::ScopedArPose scopedArPose(ar_session_);
    ArAugmentedImage_getCenterPose(ar_session_, image,
                                   scopedArPose.GetArPose());

    ArAnchor* image_anchor = nullptr;
    const ArStatus status = ArTrackable_acquireNewAnchor(
        ar_session_, ar_trackable, scopedArPose.GetArPose(), &image_anchor);

    // For example, you can now render content at the image anchor, choosing
    // content based on the image index (or name).
  }
}

Supporting different use cases

When ARCore detects an Augmented Image, it creates a Trackable for that Augmented Image and sets ArTrackingState to TRACKING and ArAugmentedImageTrackingMethod to FULL_TRACKING. When the tracked image moves out of camera view, ARCore continues to set ArTrackingState to TRACKING, but changes the ArAugmentedImageTrackingMethod to LAST_KNOWN_POSE while continuing to provide the orientation and position of the image.

Your app should use the tracking state and tracking method differently depending on the intended use case.

  • Fixed images. Most use cases involving images that are fixed in place (that is, not expected to move) can simply use ArTrackingState to determine whether the image has been detected and whether its location is known. ArAugmentedImageTrackingMethod can be ignored.

  • Moving images. If your app needs to track a moving image, use both ArTrackingState and ArAugmentedImageTrackingMethod to determine whether the image has been detected and whether its position is known.

Use case Fixed image Moving image
Example A poster hung on a wall An advertisement on the side of a bus
The pose can be
considered valid when 		ArTrackingState == TRACKING ArTrackingState == TRACKING
and
ArAugmentedImageTrackingMethod == FULL_TRACKING