Add custom data while recording on AR Foundation targeting Android

The Recording & Playback API allows you to record a session and use it in place of a real-time camera feed. However, these recordings only contain video and sensor data. You can also add custom data to a session recording and have the data returned to you during playback, as though it is part of a camera image.

ARCore does not automatically include any custom data in recordings. Rather, it allows you to add custom data to an ARCore frame during recording, and retrieve that same data from the frame during playback. It is up to you to program the app in such a way that the user will get back the data they expect when they play their session back.

Use cases for custom data

Adding custom data to recordings expands the possibilities for your AR apps. The following are some specific use cases.

Use AR on the go

In the past, users could only access AR experiences at the right place and the right time. If they wanted to place an AR lamp in their living room, they had to physically stand at the location to see how the lamp may look there. With custom tracks, they can record their living room once and add virtual furniture to the scene whenever they feel like it.

Co-create AR experiences

Without a live session requirement, users have many more options for AR editing, allowing them to create and access unique AR content at any place and time. For example, they can record a given environment, add augmented-reality effects, and share them with friends.

Prerequisites

Make sure that you understand fundamental AR concepts and how to configure an ARCore session before proceeding.

Initialize a recording with custom data

Follow these steps to initialize a recording with custom data. To start, stop, and check a recording session, please see Record and play back an AR session.

  1. Obtain an ARCoreRecordingConfig.
  2. Create a new Track with a custom UUID. All custom data will be saved here.
  3. Add the Track to the ARCoreRecordingConfig that you created during session configuration.
// Initialize a new track with a custom UUID.
// Make sure to save the UUID because it is the ID that you will use
// to get your data back during playback.
var track = new Track {
 Id = Guid.Parse("de5ec7a4-09ec-4c48-b2c3-a98b66e71893") // from UUID generator
};

// Add the Track to the recordingConfig.
// recordingConfig must already be configured.
List<Track> tracks = new List<Track>();
tracks.Add(track);
recordingConfig.Tracks = tracks;

All new tracks are treated as separate recordings, with each recorded track occupying its own UUID.

Configure the track with additional data

Configure the track with additional data that describes the session recording. For example, you can “tag” a track by adding a note that describes the location and time at which you recorded the session: “This session was recorded at the mall in the afternoon.”

// Set additional data on this track.
// For example, describe where you recorded the session.
byte[] metadata = ...
track.Metadata = metadata;

Configure the track with a MIME type

Configure a track with a MIME type that describes the type of data recorded in the track, for compatibility with external tools.

If you do not specify a type, the data will be categorized as application/text. ARCore ignores the MIME type when reading data.

// Set a MIME type for compatibility with external tools.
track.MimeType = "text/csv";

Record custom data tracks

All custom track data is recorded onto frames. The time at which you record data onto a frame is the same time at which the data will be returned during playback. For example, if you call RecordTrackData() with the value “A” at 00:07:02, you’ll get “A” back at the 00:07:02 mark when the track is played back.

To record a custom data track, call RecordTrackData().

// Place an AR lamp in a room.
if (placeLampButtonWasPressed) {
  // Convert the lamp data into a byte array.
  var lampData = new byte[] { (byte) Lamp.FLOOR };  // a floor lamp
  recordingManager.RecordTrackData(trackGuid, lampData);
}

Play back custom data tracks

Extract custom data from a session recording during playback.

Initialize a playback

Initializing a playback with custom data is the same as initializing a playback of a regular session recording.

Return custom data

Call GetUpdatedTrackData() to retrieve the custom data recorded on a frame. It is possible to retrieve multiple track data from the same frame. For example, if you called RecordTrackData() two times on the same frame during recording, you will get back two instances of TrackData during playback.

// Fetch the data recorded on a select frame and place it in a
// container object.
var trackDataList = recordingManager.GetUpdatedTrackData(trackGuid);

Once the TrackData is in a container object, extract the bytes of custom data.

// Extract the bytes of custom data from the list of track data.
foreach (TrackData trackData in trackDataList) {
  var data = trackData.Data;
  Lamp lamp = Lamp.fromByte(data[0]); // This is the lamp!
}

What’s next