PowerShell Parsing HTML returned search404Captions.content404Title

user_8103157 0
Viewed 40

I was planning to do a loop on a series of Windows KB websites to grab the names of these KB's

I used inspect and clicked on the text that I wanted so I can see the tag/class names.

$geturl = Invoke-WebRequest 'https://support.microsoft.com/en-us/help/4537759/security-update-for-adobe-flash-player'

$geturl.ParsedHtml.getElementsByTagName('h1')|?{$_.className -like 'c-heading'} | Select *html, *text

I got the following results:

innerHTML : {{search404Captions.content404Title}}
outerHTML : {{search404Captions.content404Title}}
canHaveHTML : True
innerText : {{search404Captions.content404Title}}
outerText : {{search404Captions.content404Title}}

I was looking for it to return "Security Update for Adobe Flash Player: February 11, 2020" from this Microsoft Support Article.

Can anybody tell me what I did wrong? I tried Googling but didn't see anything useful

powershell html
1 Answers

The HTML source code you're retrieving contains placeholders for information ({{ ... }}), to be filled in later by JavaScript and a templating library - the results of which you would see interactively in the browser.

(The specific elements you're retrieving are part of a div for error handling (<div ng-controller="404PageController">), so they may not be relevant to you.)

However, the information used to instantiate the placeholders with is also part of the page, inside a <script type="text/javascript"> element, which effectively contains a piece of JSON defining all information elements.

Therefore, you can:

  • extract this JSON
  • parse it into an object graph with ConvertFrom-Json
  • and access the properties of interest.

Caveat:

The fact that you need to know a potentially page-specific specific search pattern to extract the JSON and to know that JSON's structure makes this a brittle approach, as web scraping generally is. However, it may work for similar pages on the same site, assuming they were programmed analogously.

Ultimately, the only robust way to retrieve information from sites is via web services, not via HTML source code.

# Get the target page's HTML source code:
# Note: Since we'll be doing *text parsing* below, we use Invoke-RestMethod
#       rather than Invoke-WebRequest for better performance.
$htmlSource = Invoke-RestMethod https://support.microsoft.com/en-us/help/4537759/security-update-for-adobe-flash-player

# Use the regex-based -replace operator to extract the JSON of interest.
$json = $htmlSource -replace '(?s).+\bmicrosoft\.support\.prefetchedArticle\s*=\s*\(function\(\)\s*\{\s*return\s+(.+?);;.+', '$1' -replace '\}\s*\)\s*\(\s*\)\s*'

# Parse the JSON into a [pscustomobject] object graph.
$dataProperties = $json | ConvertFrom-Json

# Extract the property containing the page title.
$dataProperties.'en-us/4537759'.details.title

Note the top-level en-us/4537759 property, which suggests language-specific behavior. You may have to use the name appropriate to your system's language.

The above outputs string Security Update for Adobe Flash Player: February 11, 2020, as intended.

user_453750
Related