The Mobile Vision API is deprecated

Add Face Tracking To Your App

Stay organized with collections Save and categorize content based on your preferences.

This page is a walkthrough of how to build an app that uses the rear facing camera to show a view of the detected faces in front of you. The screenshot below shows it in action; this was taken on a tablet that was pointed at a monitor displaying a photo. We'll show you how to track several faces simultaneously and draw a rectangle around each, indicating the approximate position, size, and face ID of each face.

If you want to follow along with the code, or just want to build and try out the app, it's available in the FaceTracker folder of our Github samples.

This tutorial will discuss:

  1. Specifying custom camera settings.
  2. Tracking multiple faces.
  3. Performance / feature trade-offs.

Creating the Face Detector Pipeline

The code for setting up and executing face tracking is in FaceTrackerActivity.java, which is the main Activity for this app. Typically the face detector is specified in the onCreate method, as shown here:

@Override
public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);

    // other stuff... 

    FaceDetector detector = new FaceDetector.Builder()
        .build(getApplicationContext());

The FaceDetector.Builder specifies the properties of the face detector and initiates its creation. In this example, we use only the default face detector settings, so there are no builder properties to specify. Given a detector, we can create an associated processor pipeline to receive detection results:

detector.setProcessor(
    new MultiProcessor.Builder<Face>()
        .build(new GraphicFaceTrackerFactory()));

Create a camera source to capture video images from the camera, and continuously stream those images into the detector and its associated processor pipeline:

mCameraSource = new CameraSource.Builder()
        .setRequestedPreviewSize(640, 480)
        .setFacing(CameraSource.CAMERA_FACING_BACK)
        .setRequestedFps(30.0f)
        .build(getApplicationContext(), detector);

In this example code, we use a camera source preview UI to display the camera video and a graphic overlay to the user. We initialize the UI and start the camera like this:

mPreview.start(mCameraSource, mGraphicOverlay);

The pipeline constructed by the code above looks like this:

Once started, the rear facing camera will continuously send preview images into the pipeline. The CameraSource component manages receiving these images and passing the images into the Detector, at a maximum rate of 30 frames/second as specified above.

Detector Settings

The Detector component receives images and runs face detection/tracking on the series of images that it receives. The following default properties were used in creating the Detector above:

  • mode = fast: This indicates that the face detector can use optimizations that favor speed over accuracy. For example, it may skip faces that aren’t facing the camera.

  • landmarks = none: No facial landmarks are required for this demo. Keeping this off makes face tracking faster.

  • prominent face only = false: This demo can track multiple faces. If only a single face is required, setting this option would make face tracking faster.

  • classifications = none: Smile and eyes open classification are not required for this demo. Keeping this off makes face tracking faster.

  • tracking = true: In this app, tracking is used to maintain a consistent ID for each face. This is shown in the graphic overlay by the ID number that is displayed. As the face moves, this identity is generally maintained. However, there are a couple of reasons why the ID may change:

    • If you see an overlay that is flickering and changing color, this indicates a detection that is near the limits of what can be detected given these settings.

    • The face becomes obstructed and/or disappears and re-enters the view. Tracking works on a continuous basis, so any period of time in which the detector is not seeing the face will reset the tracking information.

MultiProcessor

The MultiProcessor is a component for working with an arbitrary number of detected items (in this case, faces). Its use was shown in the right portion of the earlier diagram:

The FaceDetector may detect multiple faces in each image frame received from the camera. Each face corresponds to a distinct face identity, as specified by the “tracking” setting above. The multiprocessor creates a Tracker<Face> instance for each face identity that it sees. It uses the factory GraphicFaceTrackerFactory, supplied by custom code above, to create new face tracker instances as needed:

private class GraphicFaceTrackerFactory 
    implements MultiProcessor.Factory<Face> {
        @Override
        public Tracker<Face> create(Face face) {
            return new GraphicFaceTracker(mGraphicOverlay);
        }
    }

As new faces are encountered, the multiprocessor will use this factory to create a new GraphicFaceTracker instance for each face. As those faces move over time, updates are routed to each of the appropriate face tracker instances. When a face is no longer visible, the multiprocessor will dispose of its associated face tracker instance. In this way, we dynamically create/track/destroy an individual face tracker for each face that we encounter in the app.

Below is the implementation of GraphicFaceTracker, which holds the state associated with an individual face:

private class GraphicFaceTracker extends Tracker<Face> {
    // other stuff

    @Override
    public void onNewItem(int faceId, Face face) {
        mFaceGraphic.setId(faceId);
    }

    @Override
    public void onUpdate(FaceDetector.Detections<Face> detectionResults,
                         Face face) {
        mOverlay.add(mFaceGraphic);
        mFaceGraphic.updateFace(face);
    }

    @Override
    public void onMissing(FaceDetector.Detections<Face> detectionResults) {
        mOverlay.remove(mFaceGraphic);
    }

    @Override
    public void onDone() {
        mOverlay.remove(mFaceGraphic);
    }
}

Each GraphicFaceTracker instance maintains an associated FaceGraphic instance, which is a graphics object that is added as part of the overlay of the camera video. This instance is created initially when the face is first encountered, updated as the face changes, hidden when the face is temporarily missing, and is removed when the face is no longer visible.