Scraping weird table structure using beautiful soup

Question

I am trying to scrape a couple of tables from this website which are hidden under an expand button. However it looks like the table structure is really weird and I'm having trouble.

my code so far is

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
import time
import requests
from bs4 import BeautifulSoup

groups = ['Currencies', 'Commodities', 'Indices']
assets = {'Currencies': ['eur/usd','gbp/usd', 'usd/jpy'], 'Commodities': ['gold', 'silver', 'copper'], 'Indices': ['spx50', 'dj30']}


browser = webdriver.Chrome(executable_path=r'/usr/local/bin/chromedriver', options=Options())
browser.get('https://www.etoro.com/trading/fees/#cfds')
soup = BeautifulSoup(browser.page_source, 'lxml')

My goal is to get the fees from the 'Spreads' section of the site (not the 'Overnight Fees' section), for each of the asset groups in my groups variable, and for each of those groups, those individual assets in the assets section. e.g. something like this for each group

| Instrument | Fee    |
|------------|--------|
| eur/usd    | 1 Pips |
| gbp/usd    | 2 Pips |
| usd/jpy    | 1 Pips |

Here is an example of the Currencies table html from beautiful soup:










EURUSD
EUR/USD

1 Pips




USDJPY
USD/JPY

1 Pips




GBPUSD
GBP/USD

2 Pips




USDCHF
USD/CHF

1.5 Pips




NZDUSD
NZD/USD

2.5 Pips




EURGBP
EUR/GBP

1.5 Pips




EURJPY
EUR/JPY

2 Pips




GBPJPY
GBP/JPY

3 Pips




AUDJPY
AUD/JPY

2 Pips




AUDUSD
AUD/USD

1 Pips




EURCHF
EUR/CHF

5 Pips




EURAUD
EUR/AUD

7 Pips




EURCAD
EUR/CAD

7 Pips




CADJPY
CAD/JPY

6 Pips




CHFJPY
CHF/JPY

6 Pips




USDHKD
USD/HKD

5 Pips




USDZAR
USD/ZAR

50 Pips




USDRUB
USD/RUB

3 Pips




USDCNH
USD/CNH

10 Pips




AUDCHF
AUD/CHF

4 Pips




AUDCAD
AUD/CAD

4 Pips




AUDNZD
AUD/NZD

4 Pips




EURNZD
EUR/NZD

4 Pips




GBPAUD
GBP/AUD

4 Pips




GBPCHF
GBP/CHF

4 Pips




GBPNZD
GBP/NZD

4 Pips




NZDCAD
NZD/CAD

4 Pips




NZDCHF
NZD/CHF

4 Pips




NZDJPY
NZD/JPY

4 Pips




CADCHF
CAD/CHF

4 Pips




USDNOK
USD/NOK

20 Pips




USDSEK
USD/SEK

20 Pips




NOKSEK
NOK/SEK

20 Pips




EURNOK
EUR/NOK

20 Pips




EURSEK
EUR/SEK

30 Pips




USDTRY
USD/TRY

50 Pips




USDMXN
USD/MXN

20 Pips




USDSGD
USD/SGD

3 Pips




GBPCAD
GBP/CAD

4 Pips




ZARJPY
ZAR/JPY

8 Pips




EURPLN
EUR/PLN

30 Pips




USDHUF
USD/HUF

20 Pips




EURHUF
EUR/HUF

20 Pips




GBPHUF
GBP/HUF

30 Pips




CHFHUF
CHF/HUF

30 Pips




USDPLN
USD/PLN

20 Pips

my first thought is to use a for loop for each one of my groups to identify the table using soup.find, then a for loop to search for the matching tr for each asset and extracting the relevant figure from td. However the website structure is confusing me too much so I'm having trouble coding it. can anybody help please?

user_6106791 · Answer

Make your life easier and just dump the HTML to pandas. All the tables are there and you can access them either with a for loop or via indexing.

Here's how:

import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from tabulate import tabulate

options = Options()
options.headless = False
driver = webdriver.Chrome(options=options)

driver.get("https://www.etoro.com/trading/fees/#cfds")
df = pd.read_html(driver.page_source, flavor="bs4")
print(tabulate(df[-9]))
driver.close()

Sample output:

--  --------------------------------------------  ------------  ------------
 0  BTC Bitcoin                                   $ -11.813292  $ 0
 1  ETHEREUM Ethereum                             $ -0.300336   $ 0
 2  BCH Bitcoin Cash                              $ -0.138758   $ 0
 3  XRP Ripple                                    $ -0.00009    $ 0
 4  DASH Dash                                     $ -0.040001   $ 0
 5  LTC Litecoin                                  $ -0.049917   $ 0
 6  ETC Ethereum Classic                          $ -0.0023     $ 0
 7  ADA Cardano                                   $ -0.000072   $ 0
 8  MIOTA IOTA                                    $ -0.000119   $ 0
 9  XLM Stellar                                   $ -0.000051   $ 0
10  EOS EOS                                       $ -0.001041   $ 0
11  NEO NEO                                       $ -0.005731   $ 0
12  TRX TRON                                      $ -0.000011   $ 0
13  ZEC ZCASH                                     $ -0.02535    $ 0
14  BNB Binance Coin                              $ -0.014941   $ 0
15  XTZ Tezos                                     $ -0.000813   $ 0
16  LINK Chainlink                                $ -0.004082   $ 0
17  UNI Uniswap                                   $ -0.000816   $ 0
18  DOGE Dogecoin                                 $ -0.000163   $ -0.000163
19  BTCEUR Bitcoin/Euro                           $ -11.813292  $ 0
20  ETHEUR Ethereum/Euro                          $ -0.300336   $ 0

and so on ...

Note: For me, running in headless mode gives me a CAPTCHA. Hence, the head mode on.

user_7326738 · Answer

First, the code below expands all the spread listings, and then continues to click the "load more" buttons for each until all the records are displayed. However, both expanding the listing and clicking the load more will not change the overall element structure on the page, so you can skip this step and proceed to the next code block that actual grabs the data and produces the result from groups and assets:

from bs4 import BeautifulSoup as soup
from selenium import webdriver
d = webdriver.Chrome('/path/to/chromedriver')
d.get('https://www.etoro.com/trading/fees/#cfds')
d.execute_script("""
   for (var i of document.querySelectorAll('span.expand-toggle')){
      i.click()
}""")
visible = d.execute_script("""
  return Array.from(document.querySelectorAll('button.show-more')).map(function(x){
      var style = window.getComputedStyle(x);
      return style.getPropertyValue('display');
   });
""")
while 'block' in visible:
   d.execute_script("""
   for (var i of document.querySelectorAll('button.show-more')){
      i.click()
   }""")
   visible = d.execute_script("""
      return Array.from(document.querySelectorAll('button.show-more')).map(function(x){
        var style = window.getComputedStyle(x);
        return style.getPropertyValue('display');
     });
   """)

Now, grabbing the tables (expanded, if the above block was run):

import re
page = soup(d.page_source, 'html.parser')
blocks = [i for i in page.select('div.page_single_tab.tab_2 div.e-row') if (k:=i['class'][-1].split('-')[-1]).isdigit() and 3 <= int(k) <= 8]
groups = ['Currencies', 'Commodities', 'Indices']
assets = {'Currencies': ['eur/usd','gbp/usd', 'usd/jpy'], 'Commodities': ['gold', 'silver', 'copper'], 'Indices': ['spx50', 'dj30']}
vals = {i.select_one('.expand-title-1').text: \
         {j.text.lower():k.select_one('.spread_num').text 
          for k in i.select('div.expand-content tr') if (j:=k.select_one('.name'))}
       for i in blocks}
result = {a:{i:vals[a].get(i) for i in b} for a, b in assets.items()}

Output:

{'Currencies': {'eur/usd': '1', 'gbp/usd': '2', 'usd/jpy': '1'}, 'Commodities': {'gold': '45', 'silver': '5', 'copper': '2'}, 'Indices': {'spx50': None, 'dj30': '600'}}

Scraping weird table structure using beautiful soup

2 Answers