I am trying to scrape an e-commerce website using requests library only.
https://b2b.baidu.com/
Every link and every request website does comes with fid and pi parameters, like this
https://b2b.baidu.com/s?q=sfsdf&from=search&fid=0,1645713795553&pi=b2b.index.search...161635618068288
However, since website loads only original HTML content without javascript-generated content, requests.get (https://b2b.baidu.com/) response doesn't have those two parameters.
I understand that alternative is to use selenium or requests_html, but for challenge and, of course, for science I would like to try to figure out API instead and scrape using requests only.
Using Chrome dev tools, i've figured out what js scripts it loads after html, but I can't figure out how to catch the moment when those two are generated and what function does the generation. Breakpoints can't catch it, since this DOM element (basically every on the page contains fid and pi) isn't getting changed, I don't know how to find how those numbers are being made :(
Any advice on what methodology should I use? I've never had experience with actual reverse-engineering js scripts before.