How to scrape specific words from the scrolling web table?

user_13636146 0
Viewed 69

I want to scrape ticker symbol from the table and check for if certain keywords are present{"GD","Daily price"," TSI"} in the "final settlement price" row. In the table, there are multiple dynamic columns. Below is one example: I want to scrape ticker Symbol of "Ambuja Cements Futures" which is "ZACE" and store it in the list and also check if certain keywords are present in the "Final settlement Price" row. A you can see in the image ZACE

After Scrolling down to the table, You can see Final settlement price row

Here is the HTML code for the ticker symbol row:

<tr>
   <th>Ticker Symbol</th>
   <td>ZADS</td>
   <td>ZACE</td>
   <td>ZAPN</td>
   <td>ZARB</td>
   <td>ZAXS</td>
   <td>ZBJA</td>
   <td>ZBAF</td>
   <td>ZBPC</td>
   <td>ZBHA</td>
   <td>ZBHI</td>
   <td>ZBOS</td>
   <td>ZCIP</td>
   <td>ZCOA</td>
   <td>ZDRR</td>
   <td>ZEIM</td>
   <td>ZGAI</td>
   <td>ZHCL</td>
   <td>ZHDB</td>
   <td>ZHMC</td>
   <td>ZHND</td>
   <td>ZHPC</td>
   <td>ZHUV</td>
   <td>ZHDF</td>
   <td>ZICI</td>
   <td>ZIHF</td>
   <td>ZIOC</td>
   <td>ZIIB</td>
   <td>ZINF</td>
   <td>ZITC</td>
   <td>ZKMB</td>
   <td>ZLT</td>
   <td>ZLPC</td>
   <td>ZMM</td>
   <td>ZMSI</td>
   <td>ZNTP</td>
   <td>ZONG</td>
   <td>ZPWG</td>
   <td>ZRIL</td>
   <td>ZSBI</td>
   <td>ZSUN</td>
   <td>ZTCS</td>
   <td>ZTTM</td>
   <td>ZTAT</td>
   <td>ZTEC</td>
   <td>ZUTC</td>
   <td>ZUPL</td>
   <td>ZVED</td>
   <td>ZWPR</td>
   <td>ZYES</td>
   <td>ZZEE</td>
</tr>

Here is the HTML code for the Final settlement price row:

<tr>
   <th>Final Settlement Price</th>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.
      <br>
   </td>
   <td></td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.
      <br>
   </td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.
      <br>
   </td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
   <td>The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.</td>
</tr>

You can visit website for the same: https://www.sgx.com/derivatives/products/indiassf?cc=ZADS

web-scraping selenium html-table beautifulsoup python
1 Answers

The data in the table is loaded dynamically so BeautifulSoup doesn't see it. But you can use re/json/requests module to get the data:

import json
import re
import requests


url = 'https://www.sgx.com/derivatives/products/indiassf?cc=ZADS'

app_config_url = 'https://www.sgx.com/config/appconfig.js'
n = re.search(r"apiVersion: '(.*?)'", requests.get(app_config_url).text).group(1)
variables = '''{{"path":"/derivatives/products/{}","lang":"EN"}}'''.format( url.split('?')[0].split('/')[-1] )
url2 = 'https://api2.sgx.com/content-api?queryId={}:page&variables={}'.format(n, variables)

data = requests.get(url2).json()

# uncomment this to prin all data:
#print(json.dumps(data, indent=4))

for c in data['data']['route']['data']['data']['contracts']:
    if c['data']['title'] == 'Ambuja Cements Futures':
        print(c['data']['tickerSymbol'])
        print(c['data']['finalSettlementPrice'])

Prints:

ZACE
The Final Settlement Price for the Contracts will be the official closing price of the Underlying Shares on the Last Trading Day, rounded to 2 decimal places, expressed as US dollars.

EDIT: New version after site update:

import json
import requests


url = 'https://www.sgx.com/derivatives/products/indiassf?cc=ZADS'

config = requests.get('https://www.sgx.com/config/appconfig.json').json()

variables = '''{{"path":"/derivatives/products/{}","lang":"EN"}}'''.format( url.split('?')[0].split('/')[-1] )
url2 = 'https://api2.sgx.com/content-api?queryId={}:page&variables={}'.format(config['cms']['apiVersion'], variables)

data = requests.get(url2).json()

# # uncomment this to prin all data:
# print(json.dumps(data, indent=4))

for c in data['data']['route']['data']['data']['contracts']:
    if c['data']['title'] == 'Ambuja Cements Futures':
        print(c['data']['tickerSymbol'])
        print(c['data']['finalSettlementPrice'])
user_100359850
Related