Use buildings and terrain around you on Android NDK (C)

The Streetscape Geometry APIs provide the geometry of terrain, buildings, or other structures in a scene. The geometry can be used for occlusion, rendering, or placing AR content via hit-test APIs. Streetscape Geometry data is obtained through Google Street View imagery.

Set up the Geospatial API

To use Streetscape Geometry, you'll need to set up the Geospatial API in your project. Follow instructions on Enabling the Geospatial API to set up the Geospatial API.

Enable Streetscape Geometry

The Geospatial API obtains Streetscape Geometry data when the ArGeospatialMode is set to ArGeospatialModeEnabled and ArStreetscapeGeometryMode is set to ArStreetscapeGeometryModeEnabled.

// Create a session config.
ArConfig* ar_config = NULL;
ArConfig_create(ar_session, &ar_config);

// Enable the Geospatial API.
ArConfig_setGeospatialMode(ar_session, ar_config, AR_GEOSPATIAL_MODE_ENABLED);
CHECK(ArSession_configure(ar_session, ar_config) == AR_SUCCESS);

// Enable Streetscape Geometry.
ArConfig_setStreetscapeGeometryMode(ar_session, ar_config, AR_STREETSCAPE_GEOMETRY_MODE_ENABLED);
CHECK(ArSession_configure(ar_session, ar_config) == AR_SUCCESS);

// Release config resources.
ArConfig_destroy(ar_config);

Obtain Streetscape Geometry in an ARCore session

Use ArSession_getAllTrackables() and use AR_TRACKABLE_STREETSCAPE_GEOMETRY to filter results.

ArTrackableList* trackable_list = NULL;
ArTrackableList_create(ar_session, &trackable_list);
ArSession_getAllTrackables(ar_session, AR_TRACKABLE_STREETSCAPE_GEOMETRY, trackable_list);

Understand ArStreetscapeGeometry

ArStreetscapeGeometry contains information about a building:

Building LOD 1

AR_STREETSCAPE_GEOMETRY_QUALITY_BUILDING_LOD_1 consists of building footprints extruded upwards to a flat top. Building heights may be inaccurate.

Building LOD 2

AR_STREETSCAPE_GEOMETRY_QUALITY_BUILDING_LOD_2 will have higher fidelity geometry. Mesh walls and roofs will more closely match the building's shape. Smaller features like chimneys or roof vents may still poke outside of the mesh.

Understand ArMesh

ArMesh is a polygon mesh representing a surface reconstruction of the Streetscape Geometry. Each ArMesh includes a vertex buffer and index buffer:

Attach AR content to a ArStreetscapeGeometry

There are two ways to attach AR content to Streetscape Geometry:

Perform a hit-test against ArStreetscapeGeometry

ArFrame_hitTest can be used to hit-test against Streetscape Geometry. If intersections are found, ArHitResult contains pose information about the hit location as well as a reference to the ArStreetscapeGeometry which was hit. This Streetscape Geometry can be passed to ArTrackable_acquireNewAnchor() to create an anchor attached to it.

ArHitResultList *hit_result_list = NULL;
ArHitResult *hit_result = NULL;
ArHitResultList_create(ar_session, &hit_result_list);
ArHitResult_create(ar_session, &hit_result);

ArFrame_hitTestRay(ar_session, ar_frame, origin, direction, hit_result_list);
ArHitResultList_getItem(ar_session, hit_result_list, 0, hit_result);

ArAnchor *out_anchor = NULL;
ArStatus status = ArHitResult_acquireNewAnchor(ar_session, hit_result, &out_anchor);
CHECK(status == AR_SUCCESS);

Enable Geospatial Depth

Geospatial Depth combines Streetscape Geometry with local sensor input to enhance depth data. When Geospatial Depth is enabled, the output depth and raw depth images are modified to include rasterized Streetscape Geometry in addition to locally observed depth. This may improve the accuracy of poses using Depth.