With iOS11, Apple introduced the ability to integrate machine learning into mobile apps with Core ML. As promising as it sounds, it also has some limitations, let’s discover it around a face detection sample app.

Couple years ago, Apple added Metal framework for graphic optimisation and calculation performance. For iOS11, the company built Core ML on top of it as a foundation framework for machine learning.

iOS Core ML

When we talks about machine learning in iOS, it’s actually integrating trained models into mobile apps, there is no learning process here.

At the top of the stack, we have the framework Vision, based on CoreML for image analysis. That’s what I’m going to use to create a face detection iOS app, I actually don’t need to use a trained model for that.

Assuming we already have an UIImage ready to use, we are going to start by creating a request to detect faces. Vision already includes two for that, VNDetectFaceRectanglesRequestfor faces, VNDetectFaceLandmarksRequestfor face features.

We also need a request handler which require a specific image format and orientation to be executed: CIImage and CGImagePropertyOrientation.

Finally our request has to be dispatched to another queue to avoid locking the UI during the detection.

At that time here is what my code looks like

let orientation = image.coreOrientation()
guard let coreImage = CIImage(image: image) else { return }

// request
let faceLandmarksRequest = VNDetectFaceRectanglesRequest(completionHandler: { [weak self] request, error in
    // do something with the result
})

// handler
DispatchQueue.global().async {
    let handler = VNImageRequestHandler(ciImage: coreImage, orientation: orientation)
    do {
        try handler.perform([faceDetectionRequest])
    } catch {
        print("Failed to perform detection .\n\(error.localizedDescription)")
    }
}

Under my request completion handler, let’s log how many faces we detected. To do so, the result would be return as an array of VNFaceObseration.

guard let observations = request.results as? [VNFaceObservation] else {
    fatalError("unexpected result type!")
}
print("Detected \(observations.count) faces")

Now we can detect faces, by drawing a square following the position given under VNFaceObservation. One of the key to achieve it is to actually flip the position and scale it based on the image height, something I discovered on Stack Overflow.

iOS Face Detection

I created a sample iOS app for my testing following that process, available on Github.