I have an app that is attempting to webscrape GHIN.com for golfers' handicaps. After prompting the user with a WKWebView page for login, the user opens the "followed" golfers page in order for the app to scrape the web page for handicap data. I am able to retrieve HTML from the webView using:
webView.evaluateJavaScript("document.documentElement.outerHTML.toString()", completionHandler: { (html: Any?, error: Error?) in
self.HTML = html as! String
print(self.HTML)
})
This code is called whenever the URL is detected to have changed (as the user logs in and navigates to the correct page). However, the HTML saved from that process includes JS code seen near the end of the string that is to be executed during the building of the page (entire HTML):
<html lang="en">
<head>
<meta charset="utf-8" />
<link rel="shortcut icon" href="/favicon.ico" />
<link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png" />
<link rel="icon" type="image/png" sizes="32x32" href="/favicon-32x32.png" />
<link rel="icon" type="image/png" sizes="16x16" href="/favicon-16x16.png" />
<link rel="manifest" href="/site.webmanifest" />
<link rel="mask-icon" href="/safari-pinned-tab.svg" color="#5bbad5" />
<meta name="msapplication-TileColor" content="#da532c" />
<meta name="theme-color" content="#ffffff" />
<meta name="apple-itunes-app" content="app-id=491796218" />
<meta name="google-play-app" content="app-id=com.advancedmobile.android.ghin" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<meta name="theme-color" content="#000000" />
<link rel="manifest" href="/manifest.json" />
<title>GHIN - United States Golf Association Service</title>
<link href="https://fonts.googleapis.com/icon?family=Material+Icons+Outlined" rel="stylesheet" />
<link href="/static/css/2.c7bb94bc.chunk.css" rel="stylesheet" />
<link href="/static/css/main.6b12f9df.chunk.css" rel="stylesheet" />
<script async="" src="//www.googletagservices.com/tag/js/gpt.js"></script>
<script src="https://securepubads.g.doubleclick.net/gpt/pubads_impl_2020082401.js?21067294" async=""></script>
</head>
<body>
<noscript>You need to enable JavaScript to run this app.</noscript>
<div id="root">
<div></div>
<div class="s-navigation" id="header" style="top: 0px;">
<header>
<div class="header__container">
<div class="menu-ham__container">
<button class="menu-ham__trigger"><span></span><span></span><span></span></button>
<div class="menu-ham__content">
<div class="menu__links__wrapper">
<ul class="menu__links">
<li class="hide-on-tablet-and-above">
<a class="menu__link" href="/post-score"><span>Post Score</span></a>
</li>
<li class="hide-on-tablet-and-above">
<a class="menu__link is-active" href="/golfer-lookup" aria-current="page"><span>Golfer Lookup</span></a>
</li>
<li class="hide-on-tablet-and-above">
<a class="menu__link" href="/stats"><span>Stats</span></a>
</li>
<li class="hide-on-tablet-and-above">
<a class="menu__link" href="/handicap-calculator/select-course"><span>Handicap Calculator</span></a>
</li>
<li class="hide-on-tablet-and-above">
<a class="menu__link" href="/profile-handicap-history"><span>Handicap History</span></a>
</li>
<li class="order-1">
<a class="menu__link" href="/my-card"><span>My Card</span></a>
</li>
<li class="order-2">
<a class="menu__link" target="_blank" href="https://www.usga.org/content/usga/home-page/handicapping.html"><span>WHS Changes</span></a>
</li>
<li class="order-3">
<a class="menu__link" href="/ghin-associations"><span>GHIN Associations</span></a>
</li>
<li class="order-4">
<a class="menu__link" href="/about-us"><span>About Ghin</span></a>
</li>
<li class="order-5">
<a class="menu__link" href="/contact-us"><span>Contact Us</span></a>
</li>
<li class="order-6">
<a class="menu__link" target="_blank" href="http://www.usga.org/handicapping/ghin-faqs.html"><span>FAQs</span></a>
</li>
</ul>
<ul class="menu__links hide-on-tablet-and-above">
<li>
<a class="menu__link" href="/?logout"><span>Log Out</span></a>
</li>
</ul>
</div>
</div>
</div>
<span role="button" tabindex="0" class="usga_logo">
golf handicap <br />
information network
</span>
<div class="user-menu__container">
<button class="user-menu__trigger"><span>username</span><i class="material-icons-outlined more">expand_more</i><i class="material-icons-outlined less">expand_less</i></button>
<div class="user-menu__content">
<ul class="menu__links compact">
<li>
<a class="menu__link" href="/?logout"><span>Log Out</span></a>
</li>
</ul>
</div>
</div>
</div>
</header>
<nav class="navigation">
<div class="container">
<ul class="nav__links">
<li><a class="nav__link" href="/post-score">Post Score</a></li>
<li><a class="nav__link is-active" href="/golfer-lookup" aria-current="page">Golfer Lookup</a></li>
<li><a class="nav__link" href="/stats">Stats</a></li>
<li><a class="nav__link" href="/handicap-calculator">Handicap Calculator</a></li>
<li><a class="nav__link" href="/profile-handicap-history">Handicap History</a></li>
</ul>
</div>
</nav>
</div>
<div class="s-navigation-placeholder" style="margin-top: 0px;"></div>
<div class="container has-adds">
<section class="golfer_lookup_section">
<div id="div-gpt-ad-1539376483266-0" class="promo_google_banner"><div style="width: 728px; height: 90px;"></div></div>
<div class="section_header">
<h3>Golfer Lookup</h3>
<ul class="tabs rounded">
<li><a href="/golfer-lookup/following" aria-current="page" class="is-active">Following</a></li>
<li><a href="/golfer-lookup/my-clubs">My Clubs</a></li>
<li><a href="/golfer-lookup/all-golfers">All Golfers</a></li>
</ul>
</div>
<div class="panel">
<div style="display: flex; justify-content: center; align-items: center; padding-top: 30px; padding-bottom: 15px;"><div class="loader"></div></div>
</div>
</section>
</div>
<footer>
<div class="footer__container">
<div class="information__container">
<div class="partners">
<h4 class="partners__title">Usga Partners</h4>
<div class="partners__list">
<a class="partner__item american-express" href="https://global.americanexpress.com/entertainment/home/" target="_blank" rel="noopener noreferrer">
<img alt="logo_american_express" class="partner__logo" src="/img/logos/logo_american_express.png" />
</a>
<a class="partner__item cisco" href="https://www.cisco.com/" target="_blank" rel="noopener noreferrer"><img alt="logo_cisco" class="partner__logo fit-size" src="/img/logos/logo_cisco.svg" /></a>
<a
class="partner__item delloite"
href="https://www2.deloitte.com/us/en/pages/consulting/articles/united-states-golf-association.html?id=us:2pm:3pt:lookagain:awa:cons:060116:usga"
target="_blank"
rel="noopener noreferrer"
>
<img alt="logo_delloite" class="partner__logo fit-size" src="/img/logos/logo_delloite.svg" />
</a>
<a class="partner__item lexus" href="https://www.lexus.com/" target="_blank" rel="noopener noreferrer"><img alt="logo_lexus" class="partner__logo lexus" src="/img/logos/logo_lexus.png" /></a>
<a class="partner__item rolex" href="https://www.rolex.com/world-of-rolex/golf.html" target="_blank" rel="noopener noreferrer">
<img alt="logo_rolex" class="partner__logo" src="/img/logos/logo_rolex.svg" />
</a>
</div>
</div>
<div class="mobile__app">
<h4 class="app__title">Ghin Mobile App</h4>
<div class="app__details">
<span class="app__logo"></span>
<ul class="app__stores">
<li><a class="store app-store" rel="noopener noreferrer" target="_blank" href="https://itunes.apple.com/us/app/ghin-mobile/id491796218?mt=8"> </a></li>
<li><a class="store google-play" rel="noopener noreferrer" target="_blank" href="https://play.google.com/store/apps/details?id=com.advancedmobile.android.ghin&hl=en_US"> </a></li>
</ul>
</div>
</div>
</div>
<ul class="footer__links">
<li><a class="footer__link" href="https://www.usga.org/content/usga/home-page/handicapping.html" target="_blank" rel="noopener noreferrer">WHS CHANGES</a></li>
<li><a class="footer__link" href="http://www.usga.org/handicapping/ghin-faqs.html" target="_blank" rel="noopener noreferrer">FAQs</a></li>
<li><a class="footer__link" href="/contact-us">CONTACT US</a></li>
<li><a class="footer__link" href="https://www.usga.org/content/usga/home-page/Handicap-ghin/ghin-privacy-policy.html" target="_blank" rel="noopener noreferrer">PRIVACY POLICY</a></li>
<li><a class="footer__link" href="https://www.usga.org/content/usga/home-page/Handicap-ghin/ghin-terms-of-service.html" target="_blank" rel="noopener noreferrer">TERMS OF SERVICE</a></li>
</ul>
<div class="copyright__container"><p>© 2020 United States Golf Association. All Rights Reserved.</p></div>
</div>
</footer>
<button id="back-top-button" class="hidden" title="Go to top"></button>
</div>
<script async="" src="https://www.google-analytics.com/analytics.js"></script>
<script>
!(function (e) {
function r(r) {
for (var n, i, l = r[0], f = r[1], a = r[2], c = 0, s = []; c < l.length; c++) (i = l[c]), Object.prototype.hasOwnProperty.call(o, i) && o[i] && s.push(o[i][0]), (o[i] = 0);
for (n in f) Object.prototype.hasOwnProperty.call(f, n) && (e[n] = f[n]);
for (p && p(r); s.length; ) s.shift()();
return u.push.apply(u, a || []), t();
}
function t() {
for (var e, r = 0; r < u.length; r++) {
for (var t = u[r], n = !0, l = 1; l < t.length; l++) {
var f = t[l];
0 !== o[f] && (n = !1);
}
n && (u.splice(r--, 1), (e = i((i.s = t[0]))));
}
return e;
}
var n = {},
o = { 1: 0 },
u = [];
function i(r) {
if (n[r]) return n[r].exports;
var t = (n[r] = { i: r, l: !1, exports: {} });
return e[r].call(t.exports, t, t.exports, i), (t.l = !0), t.exports;
}
(i.m = e),
(i.c = n),
(i.d = function (e, r, t) {
i.o(e, r) || Object.defineProperty(e, r, { enumerable: !0, get: t });
}),
(i.r = function (e) {
"undefined" != typeof Symbol && Symbol.toStringTag && Object.defineProperty(e, Symbol.toStringTag, { value: "Module" }), Object.defineProperty(e, "__esModule", { value: !0 });
}),
(i.t = function (e, r) {
if ((1 & r && (e = i(e)), 8 & r)) return e;
if (4 & r && "object" == typeof e && e && e.__esModule) return e;
var t = Object.create(null);
if ((i.r(t), Object.defineProperty(t, "default", { enumerable: !0, value: e }), 2 & r && "string" != typeof e))
for (var n in e)
i.d(
t,
n,
function (r) {
return e[r];
}.bind(null, n)
);
return t;
}),
(i.n = function (e) {
var r =
e && e.__esModule
? function () {
return e.default;
}
: function () {
return e;
};
return i.d(r, "a", r), r;
}),
(i.o = function (e, r) {
return Object.prototype.hasOwnProperty.call(e, r);
}),
(i.p = "/");
var l = (this.webpackJsonpghin = this.webpackJsonpghin || []),
f = l.push.bind(l);
(l.push = r), (l = l.slice());
for (var a = 0; a < l.length; a++) r(l[a]);
var p = f;
t();
})([]);
</script>
<script src="/static/js/2.ee61cc35.chunk.js"></script>
<script src="/static/js/main.7596f1a6.chunk.js"></script>
</body>
I need to access the HTML that is ultimately displayed by the webView (following execution of the JS code?). Is there a term for that specific part of the HTML rendering? I'm looking to pull user data from the page. It is displayed in the app (WKWebView), but is not present in the HTML saved using the above code.