iOS application fails to respond to mouse and touch events

Viewed 48

I am developing an iOS application using Swift that incorporates web content by using WKWebView.

  • [URL Loading and Goal] Once a URL has been loaded inside this web view, I want to open a keyboard whenever the user wants to type.
  • [JavaScript Events] As browsing is based on eye coordinates, I want a JavaScript event to be triggered when the user's gaze is focused on an input field. On the basis of eye gaze coordinates, I can determine the underlying element of the webpage document. After this, I want to trigger a mouse event/touch event for clicks within the input element.

However, the system-level keyboard does not appear if I follow these steps.

Code:

touchPoint variable contains the current x and y coordinates of the eye gaze. 

@IBOutlet var webView : WKWebView!

let jsStyle = """
                function sendTouchEvent(x, y, element, eventType) {
                  const touchObj = new Touch({
                    identifier: Date.now(),
                    target: element,
                    clientX: x,
                    clientY: y,
                    radiusX: 2.5,
                    radiusY: 2.5,
                    rotationAngle: 10,
                    force: 0.5,
                  });

                  const touchEvent = new TouchEvent(eventType, {
                    cancelable: true,
                    bubbles: true,
                    touches: [touchObj],
                    targetTouches: [],
                    changedTouches: [touchObj],
                    shiftKey: true,
                  });

                  element.dispatchEvent(touchEvent);
                }
               function click(x, y)
               {
                   var ev = new MouseEvent('click', {
                       'view': window,
                       'bubbles': true,
                       'cancelable': true,
                       'screenX': x,
                       'screenY': y
                   });
                   var el = document.elementFromPoint(x, y);

                    if(el.nodeName == "INPUT") {
                           el.autofocus = true;
                           el.click();
                           el.focus();
                           sendTouchEvent(x, y, el, 'touchstart');
                    }
                   try {
                        el.dispatchEvent(ev);
                    } catch(error) {
                        console.log(error);
                    }
               }
               click(\(touchPoint.x), \(touchPoint.y)); 
           """
webView.evaluateJavaScript(jsStyle, completionHandler : { result, error in
                    
     print("@#@ Evaluate JavaScript for current click!")
} => Called when eye gaze coordinates are received. 

[Temporary Solution] In order to make this work, I created a native UITextField that acts as a proxy for the webpage element and opens the keyboard by becoming the first responder when I determine using JS that the eye gaze is focused on an input element.

My goal, however, is to remove the proxy logic and replace it with JavaScript code that opens the system-level keyboard.

0 Answers
Related