I'm trying to scrape this list of options:
from lxml import html
import requests as req
ifb_resp = req.get(
url='https://www.ifb.ir/ThirdMarket/AllUnderWrited.aspx',
headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36',
'accept-language': 'en-US,en;q=0.9,fa;q=0.8'})
tree= html.fromstring(html=ifb_resp.content)
instruments = tree.xpath('//select[@id="ContentPlaceHolder1_SymbolCombo"]/option')
a1 = instruments[1]
print(a1.text)
but the text element is in Farsi(Persian) and it comes out like this:
' اعتضاد غدÛ\x8cر1_بازار سÙ\x88Ù\x85'
I tried encoding it with 'utf-8' and got this:
b' \xc3\x98\xc2\xa7\xc3\x98\xc2\xb9\xc3\x98\xc2\xaa\xc3\x98\xc2\xb6\xc3\x98\xc2\xa7\xc3\x98\xc2\xaf \xc3\x98\xc2\xba\xc3\x98\xc2\xaf\xc3\x9b\xc2\x8c\xc3\x98\xc2\xb11_\xc3\x98\xc2\xa8\xc3\x98\xc2\xa7\xc3\x98\xc2\xb2\xc3\x98\xc2\xa7\xc3\x98\xc2\xb1 \xc3\x98\xc2\xb3\xc3\x99\xc2\x88\xc3\x99\xc2\x85'
why does it turn into binary?!!!! I'm so lost here. how do I get the text as it is on the page?
