Add custom data while recording on Android

The Recording & Playback API allows you to record a session and use it in place of a real-time camera feed. However, these recordings only contain video and sensor data. You can also add custom data to a session recording and have the data returned to you during playback, as though it is part of a camera image.

ARCore does not automatically include any custom data in recordings. Rather, it allows you to add custom data to an ARCore frame during recording, and retrieve that same data from the frame during playback. It is up to you to program the app in such a way that the user will get back the data they expect when they play their session back.

Use cases for custom data

Adding custom data to recordings expands the possibilities for your AR apps. The following are some specific use cases.

Use AR on the go

In the past, users could only access AR experiences at the right place and the right time. If they wanted to place an AR lamp in their living room, they had to physically stand at the location to see how the lamp may look there. With custom tracks, they can record their living room once and add virtual furniture to the scene whenever they feel like it.

Co-create AR experiences

Without a live session requirement, users have many more options for AR editing, allowing them to create and access unique AR content at any place and time. For example, they can record a given environment, add augmented-reality effects, and share them with friends.

Prerequisites

Make sure that you understand fundamental AR concepts and how to configure an ARCore session before proceeding.

Record with custom data

Create a session recording with custom data.

Initialize a recording with custom data

Follow these steps to initialize a recording with custom data. To start, stop, and check a recording session, please see Record and play back an AR session.

1. Obtain a RecordingConfig.
2. Create a new Track with a custom UUID. All custom data will be saved here.
3. Add the Track to the RecordingConfig that you created during session configuration.

// Initialize a new track with a custom UUID.
// Make sure to save the UUID because it is the ID that you will use
// to get your data back during playback.
UUID trackUUID = UUID.fromString("de5ec7a4-09ec-4c48-b2c3-a98b66e71893"); // from UUID generator
Track track = new Track(session).setId(trackUUID);

// Add the Track to the recordingConfig.
// recordingConfig must already be configured.
recordingConfig.addTrack(track);

// Initialize a new track with a custom UUID.
// Make sure to save the UUID because it is the ID that you will use
// to get your data back during playback.
val trackUUID = UUID.fromString("de5ec7a4-09ec-4c48-b2c3-a98b66e71893") // from UUID generator
val track = Track(session).setId(trackUUID)

// Add the Track to the recordingConfig.
// recordingConfig must already be configured.
recordingConfig.addTrack(track)

All new tracks are treated as separate recordings, with each recorded track occupying its own UUID.

Optional: Configure the track with additional data

In the case that you want to identify it later, you can configure a track with additional data that describes the session recording. For example, you can “tag” a track by adding a note that describes the location and time at which you recorded the session: “This session was recorded at the mall in the afternoon.”

// Set additional data on this track.
// For example, describe where you recorded the session.
byte[] customTrackData = "airport".getBytes(StandardCharsets.UTF_8);
track.setMetadata(ByteBuffer.wrap(customTrackData));

// Set additional data on this track.
// For example, describe where you recorded the session.
val customTrackData: ByteArray = "airport".toByteArray()
track.setMetadata(ByteBuffer.wrap(customTrackData))

Optional: Configure the track with a MIME type

If your app needs to be compatible with external tools, you can configure a track with a MIME type that describes the type of data recorded in the track. If you do not specify a type, the data will be categorized as application/text. ARCore ignores the MIME type when reading data.

// Set a MIME type for compatibility with external tools.
track.setMimeType("text/csv");

// Set a MIME type for compatibility with external tools.
track.setMimeType("text/csv")

Record custom data tracks

All custom track data is recorded onto Frames. AR sessions use session.update() to get a frame. The time at which you record data onto a frame is the same time at which the data will be returned during playback. For example, if you call recordTrackData() with the value “A” at 00:07:02, you’ll get “A” back at the 00:07:02 mark when the track is played back.

To record a custom data track, convert the data into a ByteBuffer and call recordTrackData().

// Place an AR lamp in a room.
if (placeLampButtonWasPressed) {
  Lamp lampProduct = Lamp.FLOOR; // a floor lamp
  // Convert the lamp data into a byte array.
  ByteBuffer lampData = ByteBuffer.wrap(new byte[] {(byte) lampProduct.ordinal()});
  frame.recordTrackData(trackUUID, lampData);
}

// Place an AR lamp in a room.
if (placeLampButtonWasPressed) {
  val lampProduct = Lamp.FLOOR // a floor lamp
  // Convert the lamp data into a byte array.
  val lampData = ByteBuffer.wrap(byteArrayOf(lampProduct.ordinal.toByte()))
  frame.recordTrackData(trackUUID, lampData)
}

Play back custom data tracks

Extract custom data from a session recording during playback.

Initialize a playback

Initializing a playback with custom data is the same as initializing a playback of a regular session recording.

Return custom data

Call getUpdatedTrackData() to retrieve the custom data recorded on a frame. It is possible to retrieve multiple track data from the same frame. For example, if you called recordTrackData() two times on the same frame during recording, you will get back two instances of TrackData during playback.

// Fetch the data recorded on a select frame and place it in a container object.
Collection<TrackData> trackDataList = frame.getUpdatedTrackData(trackUUID);

// Fetch the data recorded on a select frame and place it in a container object.
val trackDataList: Collection<TrackData> = frame.getUpdatedTrackData(trackUUID)

Once the TrackData is in a container object, extract the bytes of custom data.

// Extract the bytes of custom data from the list of track data.
for (TrackData trackData : trackDataList) {
  ByteBuffer bytes = trackData.getData();
  Lamp lamp = Lamp.values()[bytes.get()]; // this is the lamp!
}

// Extract the bytes of custom data from the list of track data.
for (trackData in trackDataList) {
  val bytes = trackData.data
  val lamp = Lamp.values()[bytes.get().toInt()] // this is the lamp!
}

What’s next

• Learn how to build your own app with Recording & Playback by going through the Recording & Playback codelab.