Change from IE to MS Edge Browser, VBA web Scraping

Viewed 15298

I currently use IE to do some web scraping, as you may be aware IE support will be coming to an end in September 2021. I now want to update this bit of my code to use Windows Edge, however I am unsure how to do it.

The rest of the code is fine the only bit I need to change is to use another web browser. Not everyone will have Chrome or Firefox installed so I have not gone with them. However MS Edge is updated to all window PC

Ideally i would have liked it to use Edge or IE, check which is installed on PC and use that one. However I have no idea how to do it. I have tried to google the answer, however it mentions Shells and I am not to sure how I would write a code for that.

PS I will be sending the worksheet out to other users so it needs to be able to work on their PC and not just mine.

Dim Html As HTMLDocument
Dim objIE As Object
Dim Y As Integer '''integer variable we'll use as a counter
Dim result As String ''''string variable that will hold our result link
Dim pageNumber As Long ''' Number of pages.
Dim nextPageElement As Object '''page element
Dim HtmlText As Variant '''for html data
Dim myCounter As Long '''counting the number of loops data goes to sheet10 q2

Dim wsSheet As Worksheet 
Dim wb As Workbook
    Set wb = ThisWorkbook
         Set wsSheet = wb.Sheets("Sheet2")
 Y = 0 

'''################ Internet Explorer ###################
Set objIE = New InternetExplorer 'initiating a new instance of Internet Explorer and asigning it to objIE
      objIE.Visible = True 
''' Url to NAVIGATE is taken from sheet10 as is the number of pages
    objIE.navigate Sheets("Sheet10").Range("F2").Value & Replace(Worksheets("Sheet10").Range("G2") & Range("H2").Value, " ", "+") 
Do While objIE.Busy = True Or objIE.readyState <> 4: DoEvents: Loop '''wait here a few seconds while the browser is busy

    Set Html = objIE.document
        'The Parent Class is set in Sheet10 C2
        Set elements = Html.getElementsByClassName(Sheets("Sheet10").Range("C2")) ' parent CLASS
'''FOR LOOP
    For Each element In elements

As always thanks in advance.

1 Answers

Nope.

Webdriver I believe is the way forward with Edge. Although there is good content in the comments against this question: stackoverflow.com/questions/54814411/edge-browser-automation, there is no answer given.


With Webdriver you have two initial non-mutually exclusive options:

  1. Microsoft Edge (Chromium)
  2. Microsoft Edge (EdgeHTML)

The following is quoted from the links above:

Chromium:

  • Install Microsoft Edge (Chromium). To confirm that you have Microsoft Edge (Chromium) installed, go to edge://settings/help in the browser, and verify the version number is Version 75 or later.

  • Navigate to the Microsoft Edge Driver downloads page and download the driver that matches your Edge version number (edge://settings/help)

  • Download a language-specific client driver to translate your code (Python, Java, C#, Ruby, JavaScript) into commands the Microsoft Edge Driver runs in Microsoft Edge (Chromium)

EdgeHTML:

Similar to above

  • Download the appropriate Microsoft WebDriver server for your build of Windows and Microsoft Edge (EdgeHTML).
  • Download the WebDriver language binding of your choice. All Selenium language bindings support Microsoft Edge (EdgeHTML).

If you have used Selenium in any language, including VBA, then the rest is pretty easy to follow from the given code examples.

Of course, this involves a change in language (from VBA) and this doesn't solve the real problem about writing something that can run on most people's machines without extra installs, but does look like the way forward. You could always look to a remote server install that users connect to in order to guarantee availability of software (not sure how scalable this is for many).

Related