I'm been trying to load HTML from websites that have dynamic/lazy loading. So far the only solution I have seen work is by using a WebView and then using a delay (DispatchQueue.main.asyncAfter(deadline: .now() + 2.0)) before calling the evaluateJavaScript method. See below:
import SwiftSoup
import WebKit
private let myWebView: WKWebView = {
let prefs = WKPreferences()
let config = WKWebViewConfiguration()
config.preferences = prefs
let webView = WKWebView(frame: .zero, configuration: config)
return webView
}()
var allProducts: [Element] = []
override func viewDidLoad() {
super.viewDidLoad()
myWebView.navigationDelegate = self
let myURL = URL(string: "https://shop.coles.com.au/a/national/specials/search/coles-weekly-specials?pageNumber=1")
myWebView.load(URLRequest(url: myURL!))
}
func webView(_ webView: WKWebView, didFinish navigation: WKNavigation!) {
DispatchQueue.main.asyncAfter(deadline: .now() + 2.0) {
myWebView.evaluateJavaScript("document.body.innerHTML") { [self] result, error in
guard let htmlContent = result, error == nil else {
print("error")
return
}
do {
let doc = try SwiftSoup.parse(htmlContent as! String)
allProducts = try doc.select("header").array()
} catch {
print("error")
}
}
}
}
The problem with this is that having a fixed delay is vey risky/inefficient as it may not load all data or it may waste time. I though of implementing a while loop on the code so that it repeats the evaluation until it finds the data (see blow):
func webView(_ webView: WKWebView, didFinish navigation: WKNavigation!) {
while allProducts.count < 1 {
myWebView.evaluateJavaScript("document.body.innerHTML") { [self] result, error in
guard let htmlContent = result, error == nil else {
print("error")
return
}
do {
let doc = try SwiftSoup.parse(htmlContent as! String)
allProducts = try doc.select("header").array()
} catch {
print("error")
}
}
}
}
But when I ran this the console responded very unexpectedly and I don't think it the ran the evaluation code even once. I assume that you cannot call the evaluateJavaScript method more than once on a particular load.
Does anyone know any workarounds to this code, or any other way to load dynamically loaded HTML. Any help would be highly appreciated!!!!