For iOS 14, Apple announced full support for webrtc / get user media. These features have been improved / made simpler for iOS 15. I have been experimenting with webrtc and the speech recognition API. https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecognition
I have created a webrtc based web app that works as expected in Safari. I then load the web app into a webview using WKWebView and swiftui. Next, I update the info.plist file to allow speech recognition, camera (although not needed but I read somewhere that this could help) and microphone. When I run the app, I see the speech recognition service starts. We have microphone access and nothing breaks. However, I do not get the result(event) that contains the text part of speech to text: https://developer.mozilla.org/en-US/docs/Web/API/SpeechRecognition/onresult
Code for the webview
let url: URL?
let webView: WKWebView = {
let prefs = WKWebpagePreferences()
prefs.allowsContentJavaScript = true
let webConfiguration = WKWebViewConfiguration()
webConfiguration.defaultWebpagePreferences = prefs
webConfiguration.allowsInlineMediaPlayback = true
webConfiguration.allowsPictureInPictureMediaPlayback = true
webConfiguration.mediaTypesRequiringUserActionForPlayback = .all
webConfiguration.setURLSchemeHandler(MicrophoneHandler(),
forURLScheme: "Start")
return WKWebView(
frame: .zero,
configuration: webConfiguration
)
}()
func makeUIView(context: Context) -> WKWebView {
return webView
}
func updateUIView(_ uiView: WKWebView, context: Context) {
guard let myURL = url else {
return
}
self.webView.loadFileURL(myURL,
allowingReadAccessTo: myURL)
}
func getMicDetails(){
if #available(iOS 15.0, *) {
print("Microphone state: \(self.webView.microphoneCaptureState.rawValue)")
} else {
print("Feature only available for iOS 15*")
}
}
class MicrophoneHandler: NSObject, WKURLSchemeHandler {
var task: WKURLSchemeTask?
private let audioEngine = AVAudioEngine()
func webView(_ webView: WKWebView, start urlSchemeTask: WKURLSchemeTask) {
// Access microphone here
self.task = urlSchemeTask
// Get the singleton instance.
let recordingSession = AVAudioSession.sharedInstance()
do {
try recordingSession.setCategory(.playAndRecord, mode: .default)
try recordingSession.setActive(true)
recordingSession.requestRecordPermission() { allowed in
DispatchQueue.main.async {
if allowed {
print("Mic allowed")
} else {
print("Mic denied")
}
}
}
} catch {
print("Mic access failed")
}
print("Access mic here")
}
func webView(_ webView: WKWebView, stop urlSchemeTask: WKURLSchemeTask) {
// Do stuff here
}
}}
I have tested this with iOS 14.5 and iOS 15. I get the same behavior for both. Any suggestions? any ideas what I am missing?