Realistically light virtual objects in a scene

Learn how to use Lighting Estimation in your own apps.

Prerequisites

Make sure that you understand fundamental AR concepts and how to configure an ARCore session before proceeding.

Configure the API once per session with the appropriate mode

Configure Lighting Estimation once per session for the mode you want to use.

Java

// Configure the session with the Lighting Estimation API in ENVIRONMENTAL_HDR mode.
Config config = session.getConfig();
config.setLightEstimationMode(LightEstimationMode.ENVIRONMENTAL_HDR);
session.configure(config);

// Configure the session with the Lighting Estimation API in AMBIENT_INTENSITY mode.
Config config = session.getConfig();
config.setLightEstimationMode(LightEstimationMode.AMBIENT_INTENSITY);
session.configure(config);

// Configure the session with the Lighting Estimation API turned off.
Config config = session.getConfig();
config.setLightEstimationMode(LightEstimationMode.DISABLED);
session.configure(config);

Kotlin

// Configure the session with the Lighting Estimation API in ENVIRONMENTAL_HDR mode.
Config config = session.config
config.lightEstimationMode = LightEstimationMode.ENVIRONMENTAL_HDR
session.configure(config)

// Configure the session with the Lighting Estimation API in AMBIENT_INTENSITY mode.
Config config = session.config
config.lightEstimationMode = LightEstimationMode.AMBIENT_INTENSITY
session.configure(config)

// Configure the session with the Lighting Estimation API turned off.
Config config = session.config
config.lightEstimationMode = LightEstimationMode.DISABLED
session.configure(config)

Configure ENVIRONMENTAL_HDR mode

To configure ENVIRONMENTAL_HDR mode, get the light estimate for each frame, then get the environmental HDR lighting components you want to use.

Java

void update() {
  // Get the current frame.
  Frame frame = session.update();

  // Get the light estimate for the current frame.
  LightEstimate lightEstimate = frame.getLightEstimate();

  // Get intensity and direction of the main directional light from the current light estimate.
  float[] intensity = lightEstimate.getEnvironmentalHdrMainLightIntensity(); // note - currently only out param.
  float[] direction = lightEstimate.getEnvironmentalHdrMainLightDirection();
  app.setDirectionalLightValues(intensity, direction); // app-specific code.

  // Get ambient lighting as spherical harmonics coefficients.
  float[] harmonics = lightEstimate.getEnvironmentalHdrAmbientSphericalHarmonics();
  app.setAmbientSphericalHarmonicsLightValues(harmonics); // app-specific code.

  // Get HDR environmental lighting as a cubemap in linear color space.
  Image[] lightmaps = lightEstimate.acquireEnvironmentalHdrCubeMap();
  for (int i = 0; i < lightmaps.length /*should be 6*/; ++i) {
    app.uploadToTexture(i, lightmaps[i]);  // app-specific code.
  }
}

Kotlin

fun update() {
  // Get the current frame.
  val frame = session.update()

  // Get the light estimate for the current frame.
  val lightEstimate = frame.lightEstimate

  // Get intensity and direction of the main directional light from the current light estimate.
  val intensity = lightEstimate.environmentalHdrMainLightIntensity
  val direction = lightEstimate.environmentalHdrMainLightDirection
  app.setDirectionalLightValues(intensity, direction) // app-specific code.

  // Get ambient lighting as spherical harmonics coefficients.
  val harmonics = lightEstimate.environmentalHdrAmbientSphericalHarmonics
  app.ambientSphericalHarmonicsLightValues = harmonics // app-specific code.

  // Get HDR environmental lighting as a cubemap in linear color space.
  val lightMaps = lightEstimate.acquireEnvironmentalHdrCubeMap();
  for ((index, lightMap) in lightMaps.withIndex()) { // 6 maps total.
    app.uploadToTexture(index, lightMap); // app-specific code.
  }
}

Configure AMBIENT_INTENSITY mode

If you're planning to use the color correction component of AMBIENT_INTENSITY mode, first avoid allocation of color correction on every frame by reusing a shared allocation.

Java

 // Avoid allocation on every frame.
float[] colorCorrection = new float[4];

Kotlin

val colorCorrection = floatArrayOf(0.0f, 0.0f, 0.0f, 0.0f)

Get the light estimate for each frame, and then get ambient intensity components you want to use.

Java

void update() {
  // Get the current frame.
  Frame frame = session.update();

  // Get the light estimate for the current frame.
  LightEstimate lightEstimate = frame.getLightEstimate();

  // Get the pixel intensity of AMBIENT_INTENSITY mode.
  float pixelIntensity = lightEstimate.getPixelIntensity();

  // Read the pixel color correction of AMBIENT_INTENSITY mode into colorCorrection.
  lightEstimate.getColorCorrection(colorCorrection, 0);
}

Kotlin

fun update() {
    // Get the current frame.
  val frame = session.update()

  // Get the light estimate for the current frame.
  val lightEstimate = frame.lightEstimate

  // Get the pixel intensity of AMBIENT_INTENSITY mode.
  val pixelIntensity = lightEstimate.pixelIntensity

  // Read the pixel color correction of AMBIENT_INTENSITY mode into colorCorrection.
  lightEstimate.getColorCorrection(colorCorrection, 0)
}

Ensuring energy conservation with Environmental HDR APIs

Energy conservation is the principle that light reflected from a surface will never be more intense than it was before it hit the surface. This rule is enforced in physically-based rendering, but is usually omitted from legacy rendering pipelines used in video games and mobile apps.

If you're using a physically-based rendering pipeline with Environmental HDR light estimation, simply ensure physically-based materials are used in your virtual objects.

If you aren't using a physically-based pipeline, however, you have a couple of options:

  • The most ideal solution for this is to migrate to a physically-based pipeline.

  • If that isn't possible, however, a good workaround is to multiply the albedo value from a non-physically-based material by an energy conservation factor. This can make sure at least the BRDF shading model can be converted into physically-based. Each BRDF has a different factor -- for example, for a diffuse reflection it is 1/Pi.