Record and play back an AR session on AR Foundation targeting Android

The Recording & Playback API enables you to record video and AR data once within a given environment and use that content to replace a live camera session.

Prerequisites

Make sure that you understand fundamental AR concepts and how to configure an ARCore session before proceeding.

Compatibility with other ARCore APIs

Due to the way session data is processed, ARCore APIs may produce different results during playback than observed during recording. They may also produce different results during subsequent playback sessions. For example, the number of detected trackables, the precise timing of their detection, and their poses over time may be different during playback.

Compatibility with Cloud Anchors

You can host and resolve Cloud Anchors while recording or playing back a session.

Recording

Start, stop, and check the status of an ARCore session recording.

Record an ARCore session

To record an ARCore session, configure the session and provide an MP4 URI for the recording. Call ARRecordingManager.StartRecording() before resuming the session. Recording automatically starts when the session resumes. To automatically stop recording when the session is paused, call ARRecordingConfig.AutoStopOnPause. To record a partial session, call ARRecordingManager.StartRecording() while the session is running.

ARCoreRecordingConfig recordingConfig = ScriptableObject.CreateInstance<ARCoreRecordingConfig>();
Uri datasetUri = new System.Uri("file:///uri/for/dataset.mp4");
recordingConfig.Mp4DatasetUri = datasetUri.AbsoluteUri;

recordingManager.StartRecording(recordingConfig);

Stop a session recording

To stop recording without pausing the currently running AR session, call ARRecordingManager.StopRecording().

recordingManager.StopRecording();

Check recording status

ARRecordingManager.RecordingStatus can be used at any time to determine the current recording status.

Debug.Log("Current Recording Status: " + recordingManager.RecordingStatus);

Playback

Play back previously recorded AR sessions. Sessions play back in real time, and session playback or speed cannot be adjusted.

Playback a previously recorded session

To play back a previously recorded session, call ARPlaybackManager.SetPlaybackDatasetUri() and provide a URI for the dataset you wish to play back. You must pause the session to use this method. Resume the session for the change to take effect.

Once playback has started due to resuming the session, pausing the session by disabling theARSession will suspend processing of all camera image frames and any other recorded sensor data in the dataset. Camera image frames and sensor frame data that is discarded in this way will not be reprocessed when the session is again resumed by resuming the session. AR tracking for the session will generally suffer due to the gap in processed data.

// Disable the ARSession to pause the current AR session.
session.enabled = false;

// In the next frame, provide a URI for the dataset you wish to play back.
Uri datasetUri = new System.Uri("file:///uri/for/dataset.mp4");
playbackManager.SetPlaybackDatasetUri(datasetUri);

// In the frame after that, re-enable the ARSession to resume the session from
// the beginning of the dataset.
session.enabled = true;

Known issue and workaround

There is a known issue where calls to ARPlaybackManager.SetPlaybackDatasetUri() return ErrorPlaybackFailed. This happens because it can take several frames for a session to be paused. If ARPlaybackManager.SetPlaybackDatasetUri() is called before the session has been paused, it will not be able to access the session, so it will return an error.

The following code can be used as a workaround solution.

// Workaround for known issue where `playbackManager.SetPlaybackDatasetUri()`
// returns `ErrorPlaybackFailed` because it can take several frames for a
// session to be paused.

// Reference to the ARSession component in the scene.
ARSession session;

void PlaybackDataset()
{
    setPlaybackDataset = true;

    // Pause the current AR session.
    session.enabled = false;

    // Set a timeout for retrying playback retrieval.
    timeout = 10f;
}

// Next frame
void Update()
{
    ...

    if (setPlaybackDataset)
    {
        PlaybackResult result = playbackManager.SetPlaybackDatasetUri(datasetUri);
        if (result == PlaybackResult.ErrorPlaybackFailed || result == PlaybackResult.SessionNotReady)
        {
            // Try to set the dataset again in the next frame.
            timeout -= Time.deltaTime;
        }
        else
        {
            // Do not set the timeout if the result is something other than ErrorPlaybackFailed.
            timeout = -1f;
        }

        if (timeout < 0.0f)
        {
            setPlaybackDataset = false;
            // If playback is successful, proceed as usual.
            // If playback is not successful, handle the error appropriately.
        }
    }

    ...
}

Stop a playback

To stop a playback, call ARPlaybackManager.SetPlaybackDatasetUri() and set the dataset URI to null.

// Disable the ARSession to pause the current AR session.
session.enabled = false;

// In the next frame, unset the playback dataset URI.
playbackManager.SetPlaybackDatasetUri(null);

// In the frame after that, re-enable the ARSession to resume the session using
// the device camera and other sensors.
session.enabled = true;

Restart playback from the beginning

To restart a playback from the beginning of the dataset, call ARPlaybackManager.SetPlaybackDatasetUri() and specify the same MP4 recording before resuming the session.

// Disable the ARSession to pause the current AR session.
session.enabled = false;

// In the next frame, specify the same dataset URI.
playbackManager.SetPlaybackDatasetUri(datasetUri); // Same URI that was previously set.

// In the frame after that, re-enable the ARSession to resume the session from
// the beginning of the dataset.
session.enabled = true;

Play back a different session

To play back a different dataset, pause the session and specify the new dataset before resuming the session.

// Disable the ARSession to pause the current AR session.
session.enabled = false;

// In the next frame, specify a new dataset URI.
Uri newDatasetUri = new System.Uri("file:///uri/for/different/dataset.mp4");
playbackManager.SetPlaybackDatasetUri(newDatasetUri); // Different URI than was previously set.

// In the frame after that, re-enable the ARSession to resume the session from
// the beginning of the new dataset.
session.enabled = true;

Check playback status

ARPlaybackManager.PlaybackStatus can be used at any time to determine the current playback status.

Debug.Log("Current Playback Status: " + playbackManager.PlaybackStatus);

What’s next