How do I extract specific text from an image using a UITextField in Swift?

Question

I am using the Vision framework and I want to be able to use a UITextField to find a specific word in a picture. For example let's say I type in the word black in the text field and I want it to detect that in the picture I have. How would I do that? Im using Vision framework and I figured out how to detect the texts but stuck on the part where I can detect the user inputed word in the text field.

        func startTextDetection() {

       let textRequest = VNDetectTextRectanglesRequest(completionHandler: self.detectTextHandler)
       let request = VNRecognizeTextRequest(completionHandler: self.detectTextHandler)

        request.recognitionLevel = .fast
        textRequest.reportCharacterBoxes = true
        self.requests = [textRequest]

    }

    func detectTextHandler(request: VNRequest, error: Error?) {
        guard let observations = request.results else {
            print("no result")
            return
        }

        let result = observations.map({$0 as? VNTextObservation})

        DispatchQueue.main.async() {
            self.previewView.layer.sublayers?.removeSubrange(1...)
            for region in result {
                guard let rg = region else {
                    continue
                }

                self.highlightWord(box: rg)
                if let boxes = region?.characterBoxes {
                    for characterBox in boxes {
                        self.highlightLetters(box: characterBox)
                }
            }
        }
    }
}

     //when user presses search will search for text in pic. 
func textFieldShouldReturn(_ searchTextField: UITextField) -> Bool {
    searchTextField.resignFirstResponder()
    startTextDetection()

    return true
}

user_7757584 · Answer

You should watch the latest WWDC on Vision framework. Basically, from iOS 13 the VNRecognizeTextRequest returns the text and also the bounding box of the text in the image. The code can be something like this:

func startTextDetection() {
    let request = VNRecognizeTextRequest(completionHandler: self.detectTextHandler)
    request.recognitionLevel = .fast
    self.requests = [request]
}

private func detectTextHandler(request: VNRequest, error: Error?) {
    guard let observations = request.results as? [VNRecognizedTextObservation] else {
        fatalError("Received invalid observations")
    }
    for lineObservation in observations {
        guard let textLine = lineObservation.topCandidates(1).first else {
            continue
        }

        let words = textLine.string.split{ $0.isWhitespace }.map{ String($0)}
        for word in words {
            if let wordRange = textLine.string.range(of: word) {
                if let rect = try? textLine.boundingBox(for: wordRange)?.boundingBox {
                     // here you can check if word == textField.text
                     // rect is in image coordinate space, normalized with origin in the bottom left corner
                }
            }
        }
   }
}

How do I extract specific text from an image using a UITextField in Swift?

1 Answers