The Mobile Vision API is deprecated

Track Faces and Barcodes

This page is a walkthrough of how to build an app that uses the rear facing camera to show a view of the detected faces and barcodes in front of you. The screenshot below shows it in action; this was taken on a tablet that was pointed at a monitor displaying several photos. We'll show you how to track all of the barcodes and draw a rectangle around each, indicating the value, position, and size. Also, we draw an oval around each detected face, indicating the face ID (assigned sequentially), position, and size.

If you want to follow along with the code, or just want to build and try out the app, it's available in the multi-tracker folder of our Github samples.

This tutorial will discuss:

  1. Create barcode and face detectors.
  2. Use a MultiDetector to combine the barcode and face detectors.
  3. Query the detector operational status.
  4. Camera setup for combined detection.
  5. Create separate graphics for barcodes and faces.

Creating a Barcode Detector

Create a barcode detector to track barcodes:

BarcodeDetector barcodeDetector = new BarcodeDetector.Builder(context).build();

By default, the barcode detector will look for all supported barcode formats.

Alternatively, you can specify a subset of formats to detect via the BarcodeDetector.Builder.setFormats method. Reducing the number of formats to detect will make the barcode detector faster.

Creating a MultiProcessor for Managing Detected Barcodes

You must provide a TrackerFactory to create a new Tracker instance for each barcode. For this example, BarcodeTrackerFactory creates a BarcodeGraphic that overlays the bounding box and value for each barcode result:

BarcodeTrackerFactory barcodeFactory = new BarcodeTrackerFactory(mGraphicOverlay);
barcodeDetector.setProcessor(
    new MultiProcessor.Builder<>(barcodeFactory).build());

Conceptually, this creates a portion of the detector processing pipeline which looks like this:

This portion of the pipeline works as follows:

  1. The barcode detector detects barcodes and creates a collection of barcode instances.

  2. A multi-processor instance keeps track of each barcode that is currently active. It uses a factory to create a new graphic tracker instance per barcode.

  3. As barcodes are tracked across video frames, the multi-processor sends updates to the corresponding barcode tracker instances. In this case, barcode updates are simply the position and size of the barcode within the video frame.

Creating a Face Detector

Create a face detector to track faces:

FaceDetector faceDetector = new FaceDetector.Builder(context).build();

See Face API Tutorial: Face Tracker for more information on the default face detection settings that are implied by creating the face detector in this way.

Creating a MultiProcessor for Managing Detected Faces

An associated multi-processor instance is set to receive the face detection results. A factory is used by the multi-processor to create a separate tracker instance for each face:

FaceTrackerFactory faceFactory = new FaceTrackerFactory(mGraphicOverlay);
faceDetector.setProcessor(
    new MultiProcessor.Builder<>(faceFactory).build());

Conceptually, this creates a portion of the detector processing pipeline which looks like this:

This portion of the pipeline works as follows:

  1. The face detector detects faces and creates a collection of face instances.

  2. A multi-processor instance keeps track of each face that is currently active. It uses a factory to create a new graphic tracker instance per face.

  3. As faces are tracked across video frames, the multi-processor sends updates to the corresponding face tracker instances. In this case, face updates are simply the position and size of the face within the video frame. But in other cases, this could also include facial landmarks and smiling / eyes open classification.

Using a MultiDetector to Combine the Barcode and Face Detectors

A multi-detector is used to group the two detectors together as one detector. All images received by this detector from the camera will be delegated to the underlying barcode detector and face detector.

MultiDetector multiDetector = new MultiDetector.Builder()
    .add(barcodeDetector)
    .add(faceDetector)
    .build();

Conceptually, this unifies the barcode and face portions of the pipeline to look like this:

The multi-detector will receive a series of frames from the camera source. Each frame is submitted to the barcode detector and the face detector, resulting in both barcode detection and face detection on each frame.

Querying the Detector Operational Status

The first time that an app using barcode and/or face APIs is installed on a device, GMS will download a libraries to the device in order to do barcode and face detection. Usually this is done by the installer before the app is run for the first time. The vision dependencies should be included in the app’s AndroidManifest.xml in the “application” expression to enable this auto installation:

<meta-data
    android:name="com.google.android.gms.vision.DEPENDENCIES"
    android:value="barcode,face" />

But if that download has not yet completed when your app starts, then the detectors above will not detect any barcodes and/or faces (at least initially). This could happen if the user is not online, if the user lacks sufficient storage space on their device, or if the download is otherwise delayed (e.g., due to a slow network).

The detectors will automatically become operational once the library download has been completed on device.

A detector’s isOperational method can be used to check if the required native library is currently available:

if (!detector.isOperational()) {
    // ...
}

Additionally, the detection results that are created for each frame will indicate whether the corresponding detector is operational, via the Detections.detectorIsOperational() method.

Your app can take action based upon the operational state of a detector (e.g., temporarily disabling certain features, or displaying a notification to the user). Until the libraries are successfully downloaded, the detectors will be “no-op” detectors -- not detecting any results.

Camera setup for Combined Detection

Create the camera source for the rear facing camera, configured to send preview frames to the multi-detector:

mCameraSource = new CameraSource.Builder(context, multiDetector)
    .setFacing(CameraSource.CAMERA_FACING_BACK)
    .setRequestedFps(15.0f)
    .build();

Note that this will use a higher resolution by default (1024x768) in comparison to the previous face tracking example. Since the barcode detector works better at higher resolutions, this will enable detecting barcodes more reliably.

However, the downside of using a higher resolution is that face detection requires more time to run. To partially compensate for this, we also reduced the frame capture rate (FPS) to 15 frames per second. This will reduce the rate of frames being sent into the pipeline.

Creating Separate Graphics for Barcodes and Faces

Each of the GraphicTracker instances created in the earlier code snippets maintains graphics on a view that is overlayed upon the video camera preview:

As mentioned earlier, factory instances are associated with each multi-processor. Each creates a GraphicTracker instance along with a specific type of graphic to display (i.e., a barcode graphic or a face graphic):

class BarcodeTrackerFactory implements MultiProcessor.Factory<Barcode> {
    ...
    public Tracker<Barcode> create(Barcode barcode) {
        BarcodeGraphic graphic = new BarcodeGraphic(mGraphicOverlay);
        return new GraphicTracker<>(mGraphicOverlay, graphic);
    }
}

class FaceTrackerFactory implements MultiProcessor.Factory<Face> {
    ...
    public Tracker<Face> create(Face face) {
        FaceGraphic graphic = new FaceGraphic(mGraphicOverlay);
        return new GraphicTracker<>(mGraphicOverlay, graphic);
    }
}

The BarcodeGraphic and FaceGraphic classes implement the specifics of how to render the graphics associated with a barcode or face instance, respectively. For example, the BarcodeGraphic draws the detected barcode’s “raw value” and bounding box:

class BarcodeGraphic extends TrackedGraphic<Barcode> {
    ...
    public void draw(Canvas canvas) {
        ...
        canvas.drawRect(rect, mRectPaint);
        canvas.drawText(barcode.rawValue, rect.left, rect.bottom, mTextPaint);
    }
}

The FaceGraphic draws the detected face ID and bounding box (as an oval):

class FaceGraphic extends TrackedGraphic<Face> {
    ...
    public void draw(Canvas canvas) {
        float cx = translateX(face.getPosition().x + face.getWidth() / 2);
        ...
        canvas.drawText("id: " + getId(), cx + ID_X_OFFSET, cy + ID_Y_OFFSET, mIdPaint);
        canvas.drawOval(left, top, right, bottom, mBoxPaint);
    }
}