How to extract results from a saved Yahoo screener using Python?

user_17656871 0
Viewed 412

The code below should provide a list of stocks from a 'saved' yahoo finance screener. I get the list in the browser but not when running the code through python. The code works fine with Yahoo default screeners, but not with the one saved by me. Any idea how I can get this code to run for a user defined screener?

error :

Yahoo works best with the latest versions of the browsers. You're using an outdated or unsupported browser and some Yahoo features may not work properly. Please update your browser version now

code :

from bs4 import BeautifulSoup
import requests
url='https://finance.yahoo.com/screener/f4d71439-ae6d-4305-9459-1059f9aca419?count=100&offset=500'
header = {'User-Agent': 's'}
response=requests.get(url,headers=header)
soup=BeautifulSoup(response.content, 'lxml')
yahoo finance beautifulsoup xml python
1 Answers

When you saved yahoo screener that's mean it's saved in your account. For first you need login to your yahoo account (see HedgeHog's comment above). The best way to get your list of stocks it's use default screener with saved configuration like price range, volume range, markets etc. The best way to do that it's:

  1. Open Developer Tools in GoogleChrome (View -> Developer -> Developer Tools).
  2. Open https://finance.yahoo.com/gainers and configure the filter and click to "Find Stocks"
  3. After search in Developer Tools you need find page which begin like: screener?crumb
  4. Click right button then Copy -> Copy all as cURL enter image description here

This link has all table in json format: enter image description here

After that you will have cURL request with your cookies, headers and other information. For convenient you can split the curl command by python functions, for example (without my cookies :) ):

def get_headers():
return {
    "authority": "query1.finance.yahoo.com",
    "accept": "*/*",
    "accept-language": "en-US,en;q=0.9",
    "content-type": "application/json",
    "cookie": "bla bla bla COOOKIEEEEEEEESSSSS",
    "origin": "https://finance.yahoo.com",
    "referer": "https://finance.yahoo.com/gainers",
    "sec-fetch-dest": "empty",
    "sec-fetch-mode": "cors",
    "sec-ch-ua": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"102\", \"Google Chrome\";v=\"102\"",
    "sec-ch-ua-mobile": "?0",
    "sec-ch-ua-platform": "macOS",
    "sec-fetch-site": "same-site",
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36"
}

def get_data(offset):
    data = json.loads('{"offset":'+str(offset)+',"size":100,"sortField":"percentchange","sortType":"DESC","quoteType":"EQUITY","query":{"operator":"AND","operands":[{"operator":"GT","operands":["percentchange",3]},{"operator":"eq","operands":["region","us"]},{"operator":"or","operands":[{"operator":"BTWN","operands":["intradaymarketcap",2000000000,10000000000]},{"operator":"BTWN","operands":["intradaymarketcap",10000000000,100000000000]},{"operator":"GT","operands":["intradaymarketcap",100000000000]},{"operator":"LT","operands":["intradaymarketcap",2000000000]}]},{"operator":"btwn","operands":["intradayprice",0.1,15]},{"operator":"or","operands":[{"operator":"EQ","operands":["exchange","NMS"]},{"operator":"EQ","operands":["exchange","NAS"]},{"operator":"EQ","operands":["exchange","NCM"]},{"operator":"EQ","operands":["exchange","BSE"]},{"operator":"EQ","operands":["exchange","NYQ"]},{"operator":"EQ","operands":["exchange","YHD"]},{"operator":"EQ","operands":["exchange","NGM"]}]}]},"userId":"USER_ID bla bla","userIdType":"guid"}')
    return data

you response will like:

response = json.loads(requests.post(url, verify=False, headers=get_headers(), data=json.dumps(get_data(offset)), timeout=30).content)

feel free to ask if you need more help

user_195065650
Related