Add custom data while recording on Android NDK

The Recording & Playback API allows you to record a session and use it in place of a real-time camera feed. However, these recordings only contain video and sensor data. You can also add custom data to a session recording and have the data returned to you during playback, as though it is part of a camera image.

ARCore does not automatically include any custom data in recordings. Rather, it allows you to add custom data to an ARCore frame during recording, and retrieve that same data from the frame during playback. It is up to you to program the app in such a way that the user will get back the data they expect when they play their session back.

Use cases for custom data

Adding custom data to recordings expands the possibilities for your AR apps. The following are some specific use cases.

Use AR on the go

In the past, users could only access AR experiences at the right place and the right time. If they wanted to place an AR lamp in their living room, they had to physically stand at the location to see how the lamp may look there. With custom tracks, they can record their living room once and add virtual furniture to the scene whenever they feel like it.

Co-create AR experiences

Without a live session requirement, users have many more options for AR editing, allowing them to create and access unique AR content at any place and time. For example, they can record a given environment, add augmented-reality effects, and share them with friends.

Prerequisites

Make sure that you understand fundamental AR concepts and how to configure an ARCore session before proceeding.

Record with custom data

Create a session recording with custom data.

Initialize a recording with custom data

Follow these steps to initialize a recording with custom data. To start, stop, and check a recording session, please see Record and play back an AR session.

  1. Obtain an ArRecordingConfig.
  2. Create a new ArTrack with a custom UUID. All custom data will be saved here.
  3. Add the ArTrack to the ArRecordingConfig that you created during session configuration.
// Initialize a new track with a custom UUID.
// Make sure to save the UUID because it is the ID that you will use to get
// your data back during playback.
ArTrack* track = NULL;
ArTrack_create(ar_session, &track);
// String from UUID generator: de5ec7a4-09ec-4c48-b2c3-a98b66e71893
uint8_t uuid_byte_array[16] = {0xde, 0x5e, 0xc7, 0xa4, 0x09, 0xec,
                               0x4c, 0x48, 0xb2, 0xc3, 0xa9, 0x8b,
                               0x66, 0xe7, 0x18, 0x93};
ArTrack_setId(ar_session, track, uuid_byte_array);

// Add the ArTrack to the recording_config.
// recording_config must already be configured.
ArRecordingConfig_addTrack(ar_session, recording_config, track);
ArTrack_destroy(track);

All new tracks are treated as separate recordings, with each recorded track occupying its own UUID.

Optional: Configure the track with additional data

In the case that you want to identify it later, you can configure a track with additional data that describes the session recording. For example, you can “tag” a track by adding a note that describes the location and time at which you recorded the session: “This session was recorded at the mall in the afternoon.”

// Set additional data on this track.
// For example, describe where you recorded the session.
uint8_t metadata_size = 4;
uint8_t metadata[5] = "HOME";
ArTrack_setMetadata(ar_session, track, metadata, metadata_size);

Optional: Configure the track with a MIME type

If your app needs to be compatible with external tools, you can configure a track with a MIME type that describes the type of data recorded in the track. If you do not specify a type, the data will be categorized as application/text. ARCore ignores the MIME type when reading data.

// Set a MIME type for compatibility with external tools.
ArTrack_setMimeType(ar_session, track, "text/csv");

Record custom data tracks

All custom track data is recorded onto ArFrames. AR sessions use ArSession_update() to get a frame. The time at which you record data onto a frame is the same time at which the data will be returned during playback. For example, if you call ArFrame_recordTrackData() with the value “A” at 00:07:02, you’ll get “A” back at the 00:07:02 mark when the track is played back.

To record a custom data track, call ArFrame_recordTrackData().

// Place an AR lamp in a room.
if (place_lamp_button_was_pressed) {
  uint8_t lamp_data[1] = {lamp_id};
  ArFrame_recordTrackData(ar_session, frame, uuid_byte_array, lamp_data,
                          /*payload_size=*/1);
}

Play back custom data tracks

Extract custom data from a session recording during playback.

Initialize a playback

Initializing a playback with custom data is the same as initializing a playback of a regular session recording.

Return custom data

Call ArFrame_getUpdatedTrackData() to retrieve the custom data recorded on an ArFrame. It is possible to retrieve multiple track data from the same frame. For example, if you called ArFrame_recordTrackData() two times on the same frame during recording, you will get back two instances of ArTrackData during playback.

// Create the container to hold the track data retrieved from the frame.
ArTrackDataList* fetched_track_data_list;
ArTrackDataList_create(ar_session, &fetched_track_data_list);

// Fetch the track data from the frame into the created container.
ArFrame_getUpdatedTrackData(ar_session, frame, uuid_byte_array,
                            fetched_track_data_list);

Once the ArTrackData is in a container object, extract the bytes of custom data.

// Fetch the size of the track data list.
int32_t fetched_track_data_list_size;
ArTrackDataList_getSize(ar_session, fetched_track_data_list,
                        &fetched_track_data_list_size);

// Iterate through the list.
for (int i = 0; i < fetched_track_data_list_size; i++) {
  ArTrackData* fetched_track_data;
  ArTrackDataList_acquireItem(ar_session, fetched_track_data_list, i,
                              &fetched_track_data);
  // Process "fetched_track_data->sample" as desired
  ArTrackData_release(fetched_track_data);
}
ArTrackDataList_destroy(fetched_track_data_list);

What’s next