Recognizing digital ink with ML Kit on iOS

With ML Kit's digital ink recognition, you can recognize text handwritten on a digital surface in hundreds of languages, as well as classify sketches.

Try it out

  • Play around with the sample app to see an example usage of this API.

Before you begin

  1. Include the following ML Kit libraries in your Podfile:

    pod 'GoogleMLKit/DigitalInkRecognition', '7.0.0'

  2. After you install or update your project's Pods, open your Xcode project using its .xcworkspace. ML Kit is supported in Xcode version 13.2.1 or greater.

You are now ready to start recognizing text in Ink objects.

Build an Ink object

The main way to build an Ink object is to draw it on a touch screen. On iOS, you can use a UIImageView along with touch event handlers which draw the strokes on the screen and also store the strokes' points to build the Ink object. This general pattern is demonstrated in the following code snippet. See the quickstart app for a more complete example, which separates the touch event handling, screen drawing, and stroke data management.

Swift

@IBOutlet weak var mainImageView: UIImageView!
var kMillisecondsPerTimeInterval = 1000.0
var lastPoint = CGPoint.zero
private var strokes: [Stroke] = []
private var points: [StrokePoint] = []

func drawLine(from fromPoint: CGPoint, to toPoint: CGPoint) {
  UIGraphicsBeginImageContext(view.frame.size)
  guard let context = UIGraphicsGetCurrentContext() else {
    return
  }
  mainImageView.image?.draw(in: view.bounds)
  context.move(to: fromPoint)
  context.addLine(to: toPoint)
  context.setLineCap(.round)
  context.setBlendMode(.normal)
  context.setLineWidth(10.0)
  context.setStrokeColor(UIColor.white.cgColor)
  context.strokePath()
  mainImageView.image = UIGraphicsGetImageFromCurrentImageContext()
  mainImageView.alpha = 1.0
  UIGraphicsEndImageContext()
}

override func touchesBegan(_ touches: Set, with event: UIEvent?) {
  guard let touch = touches.first else {
    return
  }
  lastPoint = touch.location(in: mainImageView)
  let t = touch.timestamp
  points = [StrokePoint.init(x: Float(lastPoint.x),
                             y: Float(lastPoint.y),
                             t: Int(t * kMillisecondsPerTimeInterval))]
  drawLine(from:lastPoint, to:lastPoint)
}

override func touchesMoved(_ touches: Set, with event: UIEvent?) {
  guard let touch = touches.first else {
    return
  }
  let currentPoint = touch.location(in: mainImageView)
  let t = touch.timestamp
  points.append(StrokePoint.init(x: Float(currentPoint.x),
                                 y: Float(currentPoint.y),
                                 t: Int(t * kMillisecondsPerTimeInterval)))
  drawLine(from: lastPoint, to: currentPoint)
  lastPoint = currentPoint
}

override func touchesEnded(_ touches: Set, with event: UIEvent?) {
  guard let touch = touches.first else {
    return
  }
  let currentPoint = touch.location(in: mainImageView)
  let t = touch.timestamp
  points.append(StrokePoint.init(x: Float(currentPoint.x),
                                 y: Float(currentPoint.y),
                                 t: Int(t * kMillisecondsPerTimeInterval)))
  drawLine(from: lastPoint, to: currentPoint)
  lastPoint = currentPoint
  strokes.append(Stroke.init(points: points))
  self.points = []
  doRecognition()
}

Objective-C

// Interface
@property (weak, nonatomic) IBOutlet UIImageView *mainImageView;
@property(nonatomic) CGPoint lastPoint;
@property(nonatomic) NSMutableArray *strokes;
@property(nonatomic) NSMutableArray *points;

// Implementations
static const double kMillisecondsPerTimeInterval = 1000.0;

- (void)drawLineFrom:(CGPoint)fromPoint to:(CGPoint)toPoint {
  UIGraphicsBeginImageContext(self.mainImageView.frame.size);
  [self.mainImageView.image drawInRect:CGRectMake(0, 0, self.mainImageView.frame.size.width,
                                                  self.mainImageView.frame.size.height)];
  CGContextMoveToPoint(UIGraphicsGetCurrentContext(), fromPoint.x, fromPoint.y);
  CGContextAddLineToPoint(UIGraphicsGetCurrentContext(), toPoint.x, toPoint.y);
  CGContextSetLineCap(UIGraphicsGetCurrentContext(), kCGLineCapRound);
  CGContextSetLineWidth(UIGraphicsGetCurrentContext(), 10.0);
  CGContextSetRGBStrokeColor(UIGraphicsGetCurrentContext(), 1, 1, 1, 1);
  CGContextSetBlendMode(UIGraphicsGetCurrentContext(), kCGBlendModeNormal);
  CGContextStrokePath(UIGraphicsGetCurrentContext());
  CGContextFlush(UIGraphicsGetCurrentContext());
  self.mainImageView.image = UIGraphicsGetImageFromCurrentImageContext();
  UIGraphicsEndImageContext();
}

- (void)touchesBegan:(NSSet *)touches withEvent:(nullable UIEvent *)event {
  UITouch *touch = [touches anyObject];
  self.lastPoint = [touch locationInView:self.mainImageView];
  NSTimeInterval time = [touch timestamp];
  self.points = [NSMutableArray array];
  [self.points addObject:[[MLKStrokePoint alloc] initWithX:self.lastPoint.x
                                                         y:self.lastPoint.y
                                                         t:time * kMillisecondsPerTimeInterval]];
  [self drawLineFrom:self.lastPoint to:self.lastPoint];
}

- (void)touchesMoved:(NSSet *)touches withEvent:(nullable UIEvent *)event {
  UITouch *touch = [touches anyObject];
  CGPoint currentPoint = [touch locationInView:self.mainImageView];
  NSTimeInterval time = [touch timestamp];
  [self.points addObject:[[MLKStrokePoint alloc] initWithX:currentPoint.x
                                                         y:currentPoint.y
                                                         t:time * kMillisecondsPerTimeInterval]];
  [self drawLineFrom:self.lastPoint to:currentPoint];
  self.lastPoint = currentPoint;
}

- (void)touchesEnded:(NSSet *)touches withEvent:(nullable UIEvent *)event {
  UITouch *touch = [touches anyObject];
  CGPoint currentPoint = [touch locationInView:self.mainImageView];
  NSTimeInterval time = [touch timestamp];
  [self.points addObject:[[MLKStrokePoint alloc] initWithX:currentPoint.x
                                                         y:currentPoint.y
                                                         t:time * kMillisecondsPerTimeInterval]];
  [self drawLineFrom:self.lastPoint to:currentPoint];
  self.lastPoint = currentPoint;
  if (self.strokes == nil) {
    self.strokes = [NSMutableArray array];
  }
  [self.strokes addObject:[[MLKStroke alloc] initWithPoints:self.points]];
  self.points = nil;
  [self doRecognition];
}

Note that the code snippet includes a sample function to draw the stroke into the UIImageView, which should be adapted as necessary for your application. We recommend using roundcaps when drawing the line segments so that zero length segments will be drawn as a dot (think of the dot on a lowercase letter i). The doRecognition() function is called after each stroke is written and will be defined below.

Get an instance of DigitalInkRecognizer

To perform recognition we need to pass the Ink object to a DigitalInkRecognizer instance. To obtain the DigitalInkRecognizer instance, we first need to download the recognizer model for the desired language, and load the model into RAM. This can be accomplished using the following code snippet, which for simplicity is placed in the viewDidLoad() method and uses a hardcoded language name. See the quickstart app for an example of how to show the list of available languages to the user and download the selected language.

Swift

override func viewDidLoad() {
  super.viewDidLoad()
  let languageTag = "en-US"
  let identifier = DigitalInkRecognitionModelIdentifier(forLanguageTag: languageTag)
  if identifier == nil {
    // no model was found or the language tag couldn't be parsed, handle error.
  }
  let model = DigitalInkRecognitionModel.init(modelIdentifier: identifier!)
  let modelManager = ModelManager.modelManager()
  let conditions = ModelDownloadConditions.init(allowsCellularAccess: true,
                                         allowsBackgroundDownloading: true)
  modelManager.download(model, conditions: conditions)
  // Get a recognizer for the language
  let options: DigitalInkRecognizerOptions = DigitalInkRecognizerOptions.init(model: model)
  recognizer = DigitalInkRecognizer.digitalInkRecognizer(options: options)
}

Objective-C

- (void)viewDidLoad {
  [super viewDidLoad];
  NSString *languagetag = @"en-US";
  MLKDigitalInkRecognitionModelIdentifier *identifier =
      [MLKDigitalInkRecognitionModelIdentifier modelIdentifierForLanguageTag:languagetag];
  if (identifier == nil) {
    // no model was found or the language tag couldn't be parsed, handle error.
  }
  MLKDigitalInkRecognitionModel *model = [[MLKDigitalInkRecognitionModel alloc]
                                          initWithModelIdentifier:identifier];
  MLKModelManager *modelManager = [MLKModelManager modelManager];
  [modelManager downloadModel:model conditions:[[MLKModelDownloadConditions alloc]
                                                initWithAllowsCellularAccess:YES
                                                allowsBackgroundDownloading:YES]];
  MLKDigitalInkRecognizerOptions *options =
      [[MLKDigitalInkRecognizerOptions alloc] initWithModel:model];
  self.recognizer = [MLKDigitalInkRecognizer digitalInkRecognizerWithOptions:options];
}

The quickstart apps include additional code that shows how to handle multiple downloads at the same time, and how to determine which download succeeded by handling the completion notifications.

Recognize an Ink object

Next we come to the doRecognition() function, which for simplicity is called from touchesEnded(). In other applications one might want to invoke recognition only after a timeout, or when the user pressed a button to trigger recognition.

Swift

func doRecognition() {
  let ink = Ink.init(strokes: strokes)
  recognizer.recognize(
    ink: ink,
    completion: {
      [unowned self]
      (result: DigitalInkRecognitionResult?, error: Error?) in
      var alertTitle = ""
      var alertText = ""
      if let result = result, let candidate = result.candidates.first {
        alertTitle = "I recognized this:"
        alertText = candidate.text
      } else {
        alertTitle = "I hit an error:"
        alertText = error!.localizedDescription
      }
      let alert = UIAlertController(title: alertTitle,
                                  message: alertText,
                           preferredStyle: UIAlertController.Style.alert)
      alert.addAction(UIAlertAction(title: "OK",
                                    style: UIAlertAction.Style.default,
                                  handler: nil))
      self.present(alert, animated: true, completion: nil)
    }
  )
}

Objective-C

- (void)doRecognition {
  MLKInk *ink = [[MLKInk alloc] initWithStrokes:self.strokes];
  __weak typeof(self) weakSelf = self;
  [self.recognizer
      recognizeInk:ink
        completion:^(MLKDigitalInkRecognitionResult *_Nullable result,
                     NSError *_Nullable error) {
    typeof(weakSelf) strongSelf = weakSelf;
    if (strongSelf == nil) {
      return;
    }
    NSString *alertTitle = nil;
    NSString *alertText = nil;
    if (result.candidates.count > 0) {
      alertTitle = @"I recognized this:";
      alertText = result.candidates[0].text;
    } else {
      alertTitle = @"I hit an error:";
      alertText = [error localizedDescription];
    }
    UIAlertController *alert =
        [UIAlertController alertControllerWithTitle:alertTitle
                                            message:alertText
                                     preferredStyle:UIAlertControllerStyleAlert];
    [alert addAction:[UIAlertAction actionWithTitle:@"OK"
                                              style:UIAlertActionStyleDefault
                                            handler:nil]];
    [strongSelf presentViewController:alert animated:YES completion:nil];
  }];
}

Managing model downloads

We have already seen how to download a recognition model. The following code snippets illustrate how to check whether a model has already been downloaded, or to delete a model when it is no longer needed to recover the storage space.

Check whether a model has been downloaded already

Swift

let model : DigitalInkRecognitionModel = ...
let modelManager = ModelManager.modelManager()
modelManager.isModelDownloaded(model)

Objective-C

MLKDigitalInkRecognitionModel *model = ...;
MLKModelManager *modelManager = [MLKModelManager modelManager];
[modelManager isModelDownloaded:model];

Delete a downloaded model

Swift

let model : DigitalInkRecognitionModel = ...
let modelManager = ModelManager.modelManager()

if modelManager.isModelDownloaded(model) {
  modelManager.deleteDownloadedModel(
    model!,
    completion: {
      error in
      if error != nil {
        // Handle error
        return
      }
      NSLog(@"Model deleted.");
    })
}

Objective-C

MLKDigitalInkRecognitionModel *model = ...;
MLKModelManager *modelManager = [MLKModelManager modelManager];

if ([self.modelManager isModelDownloaded:model]) {
  [self.modelManager deleteDownloadedModel:model
                                completion:^(NSError *_Nullable error) {
                                  if (error) {
                                    // Handle error.
                                    return;
                                  }
                                  NSLog(@"Model deleted.");
                                }];
}

Tips to improve text recognition accuracy

The accuracy of text recognition can vary across different languages. Accuracy also depends on writing style. While Digital Ink Recognition is trained to handle many kinds of writing styles, results can vary from user to user.

Here are some ways to improve the accuracy of a text recognizer. Note that these techniques do not apply to the drawing classifiers for emojis, autodraw, and shapes.

Writing area

Many applications have a well defined writing area for user input. The meaning of a symbol is partially determined by its size relative to the size of the writing area that contains it. For example, the difference between a lower or upper case letter "o" or "c", and a comma versus a forward slash.

Telling the recognizer the width and height of the writing area can improve accuracy. However, the recognizer assumes that the writing area only contains a single line of text. If the physical writing area is large enough to allow the user to write two or more lines, you may get better results by passing in a WritingArea with a height that is your best estimate of the height of a single line of text. The WritingArea object you pass to the recognizer does not have to correspond exactly with the physical writing area on the screen. Changing the WritingArea height in this way works better in some languages than others.

When you specify the writing area, specify its width and height in the same units as the stroke coordinates. The x,y coordinate arguments have no unit requirement - the API normalizes all units, so the only thing that matters is the relative size and position of strokes. You are free to pass in coordinates in whatever scale makes sense for your system.

Pre-context

Pre-context is the text that immediately precedes the strokes in the Ink that you are trying to recognize. You can help the recognizer by telling it about the pre-context.

For example, the cursive letters "n" and "u" are often mistaken for one another. If the user has already entered the partial word "arg", they might continue with strokes that can be recognized as "ument" or "nment". Specifying the pre-context "arg" resolves the ambiguity, since the word "argument" is more likely than "argnment".

Pre-context can also help the recognizer identify word breaks, the spaces between words. You can type a space character but you cannot draw one, so how can a recognizer determine when one word ends and the next one starts? If the user has already written "hello" and continues with the written word "world", without pre-context the recognizer returns the string "world". However, if you specify the pre-context "hello", the model will return the string " world", with a leading space, since "hello world" makes more sense than "helloword".

You should provide the longest possible pre-context string, up to 20 characters, including spaces. If the string is longer, the recognizer only uses the last 20 characters.

The code sample below shows how to define a writing area and use a RecognitionContext object to specify pre-context.

Swift

let ink: Ink = ...;
let recognizer: DigitalInkRecognizer =  ...;
let preContext: String = ...;
let writingArea = WritingArea.init(width: ..., height: ...);

let context: DigitalInkRecognitionContext.init(
    preContext: preContext,
    writingArea: writingArea);

recognizer.recognizeHandwriting(
  from: ink,
  context: context,
  completion: {
    (result: DigitalInkRecognitionResult?, error: Error?) in
    if let result = result, let candidate = result.candidates.first {
      NSLog("Recognized \(candidate.text)")
    } else {
      NSLog("Recognition error \(error)")
    }
  })

Objective-C

MLKInk *ink = ...;
MLKDigitalInkRecognizer *recognizer = ...;
NSString *preContext = ...;
MLKWritingArea *writingArea = [MLKWritingArea initWithWidth:...
                                              height:...];

MLKDigitalInkRecognitionContext *context = [MLKDigitalInkRecognitionContext
       initWithPreContext:preContext
       writingArea:writingArea];

[recognizer recognizeHandwritingFromInk:ink
            context:context
            completion:^(MLKDigitalInkRecognitionResult
                         *_Nullable result, NSError *_Nullable error) {
                               NSLog(@"Recognition result %@",
                                     result.candidates[0].text);
                         }];

Stroke ordering

Recognition accuracy is sensitive to the order of the strokes. The recognizers expect strokes to occur in the order people would naturally write; for example left-to-right for English. Any case that departs from this pattern, such as writing an English sentence starting with the last word, gives less accurate results.

Another example is when a word in the middle of an Ink is removed and replaced with another word. The revision is probably in the middle of a sentence, but the strokes for the revision are at the end of the stroke sequence. In this case we recommend sending the newly written word separately to the API and merging the result with the prior recognitions using your own logic.

Dealing with ambiguous shapes

There are cases where the meaning of the shape provided to the recognizer is ambiguous. For example, a rectangle with very rounded edges could be seen as either a rectangle or an ellipse.

These unclear cases can be handled by using recognition scores when they are available. Only shape classifiers provide scores. If the model is very confident, the top result's score will be much better than the second best. If there is uncertainty, the scores for the top two results will be close. Also, keep in mind that the shape classifiers interpret the whole Ink as a single shape. For example, if the Ink contains a rectangle and an ellipse next to each other, the recognizer may return one or the other (or something completely different) as a result, since a single recognition candidate cannot represent two shapes.