Speech recognition via webkit for iOS WKWebViews

Viewed 455

For iOS 14, Apple announced full support for webrtc / get user media. These features have been improved / made simpler for iOS 15. I have been experimenting with webrtc and the speech recognition API. https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecognition

I have created a webrtc based web app that works as expected in Safari. I then load the web app into a webview using WKWebView and swiftui. Next, I update the info.plist file to allow speech recognition, camera (although not needed but I read somewhere that this could help) and microphone. When I run the app, I see the speech recognition service starts. We have microphone access and nothing breaks. However, I do not get the result(event) that contains the text part of speech to text: https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecognition/onresult

Code for the webview

    let url: URL?
    let webView: WKWebView = {
        let prefs = WKWebpagePreferences()
        prefs.allowsContentJavaScript = true
        let webConfiguration = WKWebViewConfiguration()
        webConfiguration.defaultWebpagePreferences = prefs
        webConfiguration.allowsInlineMediaPlayback = true
        webConfiguration.allowsPictureInPictureMediaPlayback = true
        webConfiguration.mediaTypesRequiringUserActionForPlayback = .all
        webConfiguration.setURLSchemeHandler(MicrophoneHandler(),
                                             forURLScheme: "Start")
        return WKWebView(
            frame: .zero,
            configuration: webConfiguration
        )
      
   
    }()
 
    func makeUIView(context: Context) -> WKWebView {
        return webView
    }

    func updateUIView(_ uiView: WKWebView, context: Context) {
        guard let myURL = url else {
            return
        }
        self.webView.loadFileURL(myURL,
                                  allowingReadAccessTo: myURL)
    }
    
    func getMicDetails(){
        if #available(iOS 15.0, *) {
            print("Microphone state: \(self.webView.microphoneCaptureState.rawValue)")
        } else {
            print("Feature only available for iOS 15*")
        }
    }

    class MicrophoneHandler: NSObject, WKURLSchemeHandler {
        var task: WKURLSchemeTask?
        private let audioEngine = AVAudioEngine()
        
        func webView(_ webView: WKWebView, start urlSchemeTask: WKURLSchemeTask) {
            // Access microphone here
            self.task = urlSchemeTask
            // Get the singleton instance.
            let recordingSession = AVAudioSession.sharedInstance()
            do {
                   try recordingSession.setCategory(.playAndRecord, mode: .default)
                   try recordingSession.setActive(true)
                   recordingSession.requestRecordPermission() { allowed in
                       DispatchQueue.main.async {
                           if allowed {
                               print("Mic allowed")
                           } else {
                               print("Mic denied")
                           }
                       }
                   }
               } catch {
                   print("Mic access failed")
               }
            print("Access mic here")
        }
        
        func webView(_ webView: WKWebView, stop urlSchemeTask: WKURLSchemeTask) {
            // Do stuff here
        }
        
        
    }}

I have tested this with iOS 14.5 and iOS 15. I get the same behavior for both. Any suggestions? any ideas what I am missing?

0 Answers
Related