Learn how to use the Scene Semantics API in your own apps.
The Scene Semantics API enables developers to understand the scene surrounding the user, by providing ML model-based, real-time semantic information. Given an image of an outdoor scene, the API returns a label for each pixel across a set of useful semantic classes, such a sky, building, tree, road, sidewalk, vehicle, person, and more. In addition to pixel labels, the Scene Semantics API also offers confidence values for each pixel label and an easy-to-use way to query the prevalence of a given label in an outdoor scene.
From left to right, examples of an input image, the semantic image of pixel labels, and the corresponding confidence image:
Enable Scene Semantics
In a new ARCore session, check whether a user's device supports the Scene Semantics API. Not all ARCore-compatible devices support the Scene Semantics API due to processing power constraints.
To save resources, Scene Semantics is disabled by default on ARCore. Enable semantic mode to have your app use the Scene Semantics API.
In your ARCoreExtensionsConfig, set the Semantics Mode to Enabled.
If using iOS, Semantics must also be enabled in the project settings:
- Navigate to Edit > Project Settings > XR Plug-In Management > ARCore Extensions.
- Under Optional Features, select Semantics on iOS.
Obtain the semantic image
ArSemanticManager.TryGetSemanticTexture() to acquire the semantic image:
if (semanticManager.TryGetSemanticTexture(out Texture2D semanticImage))
// Use the semantic image here.
Output semantic images should be available after about 1-3 frames from the start of the session, depending on the device.
Obtain the confidence image
In addition to the semantic image, which provides a label for each pixel, the API also provides a confidence image of corresponding pixel confidence values. The confidence image is a
TextureFormat.Alpha8 image, where each pixel corresponds to a value in the range
[0, 255], corresponding to the probability associated with the semantic label for each pixel.
ArSemanticManager.TryGetSemanticConfidenceTexture() to acquire the semantic confidence image:
if (semanticManager.TryGetSemanticConfidenceTexture(out Texture2D semanticConfidenceImage))
// Use the semantic confidence image here.
Output confidence images should be available after about 1-3 frames from the start of the session, depending on the device.
Query the fraction of pixels for a semantic label
You can also query the fraction of pixels in the current frame that belong to a particular class, such as sky. This query is more efficient than returning the semantic image and performing a pixel-wise search for a specific label. The returned fraction is a float value in the range
ArSemanticManager.GetSemanticLabelFraction() to acquire the fraction for a given label:
var fraction = semanticManager.GetSemanticLabelFraction(SemanticLabel.SKY);