Can someone guide me how to collect a list of url address in the tab using python?

Viewed 73

I'm trying to collect a list of "https://..." and hope to store them in csv file. I can do them manually such as use excel, copy the urls from the website of interest and paste them one by one. But it's tedious and definitely would take lot of time.

can someone suggest and guide for a faster way?

1 Answers

If you just need the addresses quickly from one page you could run this javascript snippet document.links.forEach(link=>console.log(link.href)) in the console of your browser, this will output all of the links on that page.

If you want to use python to scrape the page I would suggest taking a look at this question on stackoverflow, this uses the beautifulsoup framework.

If there is dynamic content loaded on the page with javascript it's probably better to use something like Selenium, relevant stackoverflow question

Related