How to mimic "Save As" of a webpage to get its source code?

Question

I'm trying to download the source code of a webpage.

When I tried "View Page Source", the portion of the webpage that I'm interested in is not in it. It seems to be "hidden" in

and the actual text/formating is not there.

If I do a "Save As", I get everything, including the portion of the page that I want. The code also looks a bit different:

...

I can also see this code if I do the "Inspect Element" using Chrome.

How can I get the full source code of this webpage (not including other parts of the website), like what I can get when I "Save" the webpage? Can I do it using curl or wget?

user_9448090 · Answer

curl and wget would proably not work. Pages like this are usually loaded dynamically using javascript. One way around that is to use Selenium (or its competitors) to emulate browser behavior. There are tons of postings about this, for example here.

I haven't personally tried it, but I've recently run into Shellnium: Simple Selnium WebDriver for Bash which you may want to take a look into.

How to mimic "Save As" of a webpage to get its source code?

1 Answers