Interactive image segmentation guide for Android

The MediaPipe Interactive Image Segmenter task takes strokes indicating a location in an image, estimates the boundaries of an object at that location, and returns the segmentation for the object as image data. These instructions show you how to use the Interactive Image Segmenter with Android apps. The code example described in these instructions is available on GitHub. For more information about the capabilities, models, and configuration options of this task, see the Overview.

Code example

The MediaPipe Tasks code example is a basic implementation of a Interactive Image Segmenter app for Android. The example works with images selected from the device gallery.

You can use the app as a starting point for your own Android app, or refer to it when modifying an existing app. The Interactive Image Segmenter example code is hosted on GitHub.

Download the code

The following instructions show you how to create a local copy of the example code using the git command line tool.

To download the example code:

  1. Clone the git repository using the following command: 
    git clone https://github.com/google-ai-edge/mediapipe-samples
  2. Optionally, configure your git instance to use sparse checkout, so you have only the files for the Interactive Image Segmenter example app: 
    cd mediapipe-samples
git sparse-checkout init --cone
git sparse-checkout set examples/interactive_segmentation/android

After creating a local version of the example code, you can import the project into Android Studio and run the app. For instructions, see the Setup Guide for Android.

Key components

The following files contain the crucial code for this image segmentation example application:

Setup

This section describes key steps for setting up your development environment and code projects to use Interactive Image Segmenter. For general information on setting up your development environment for using MediaPipe tasks, including platform version requirements, see the Setup guide for Android.

Dependencies

Interactive Image Segmenter uses the com.google.mediapipe:tasks-vision library. Add this dependency to the build.gradle file of your Android app development project. Import the required dependencies with the following code:

dependencies {
    ...
    implementation 'com.google.mediapipe:tasks-vision:latest.release'
}

Model

The MediaPipe Interactive Image Segmenter task requires a trained model that is compatible with this task. For more information on available trained models for Interactive Image Segmenter, see the task overview Models section.

Select and download the model, and then store it within your project directory:

<dev-project-root>/src/main/assets

Use the BaseOptions.Builder.setModelAssetPath() method to specify the path used by the model. This method is shown in the code example in the next section.

In the Interactive Image Segmenter example code, the model is defined in the InteractiveSegmentationHelper class in the setupInteractiveSegmenter() function.

Create the task

You can use the createFromOptions function to create the task. The createFromOptions function accepts configuration options. For more information on configuration options, see Configuration options.

BaseOptions baseOptions = BaseOptions.builder()
    .setModelAssetPath("interactive_segmentation.task")
    .build();

InteractiveSegmenterOptions options = InteractiveSegmenterOptions.builder()
    .setBaseOptions(baseOptions)
    .build();

InteractiveSegmenter interactiveSegmenter =
    InteractiveSegmenter.createFromOptions(context, options);

For a more detailed example of setting up this task, see the InteractiveSegmentationHelper class setupInteractiveSegmenter() function.

Configuration options

This task has no task-specific configuration options. It only supports the BaseOptions configuration common to all tasks, which lets you specify:

  • Model asset path: The location of the pre-trained task model within your project.
  • Delegate: The processing hardware for running the task (such as CPU or GPU).

Prepare data

Interactive Image Segmenter works with images, and the task handles the data input preprocessing, including resizing, rotation and value normalization. You need to convert the input image to a com.google.mediapipe.framework.image.MPImage object before passing it to the task.

import com.google.mediapipe.framework.image.BitmapImageBuilder;
import com.google.mediapipe.framework.image.MPImage;

// Load an image on the user's device as a Bitmap object using BitmapFactory.

// Convert an Android Bitmap object to a MediaPipe Image object.
MPImage mpImage = new BitmapImageBuilder(bitmap).build();

In the Interactive Image Segmenter example code, the data preparation is handled in the InteractiveSegmentationHelper class by the segment() function.

Run the task

Call the segment function to run the prediction and generate a segmentation mask. The Interactive Image Segmenter task returns a single mask representing the identified segment region within the input image.

import com.google.mediapipe.tasks.components.containers.NormalizedKeypoint;
import com.google.mediapipe.tasks.vision.interactivesegmenter.Stroke;
import java.util.Arrays;
import java.util.List;

// Set the image to be segmented.
interactiveSegmenter.setImage(mpImage);

// Perform image segmentation on the provided image using a stroke.
List<Stroke> strokes = Arrays.asList(
    Stroke.builder()
        .setBrushMode(Stroke.BrushMode.POSITIVE)
        .setPoints(Arrays.asList(
            NormalizedKeypoint.create(0.2f, 0.5f),
            NormalizedKeypoint.create(0.5f, 0.5f)
        ))
        .setCompleted(true)
        .build()
);

MPImage mask = interactiveSegmenter.segment(strokes);

In the Interactive Image Segmenter example code, the segment functions are defined in the InteractiveSegmentationHelper.kt file.

Handle and display results

Upon running inference, the Interactive Image Segmenter task returns an MPImage object which contains the results of the segmentation task. The content of the output is a confidence mask.

The following sections further explain the output data from this task:

Confidence mask

The output for the interactive segmenter is a single-channel confidence mask containing float values between [0.0, 1.0] for each pixel of the image, with dimensions matching the input image. Higher values indicate a higher confidence that the image pixel is part of the object located at the area of interest.