How to get past the login page with Wget?

Viewed 348319

I am trying to use Wget to download a page, but I cannot get past the login screen.

How do I send the username/password using post data on the login page and then download the actual page as an authenticated user?

11 Answers

I wanted a one-liner that didn't download any files; here is an example of piping the cookie output into the next request. I only tested the following on Gentoo, but it should work in most *nix environments:

wget -q -O /dev/null --save-cookies /dev/stdout --post-data 'u=user&p=pass' 'http://example.com/login' | wget -q -O - --load-cookies /dev/stdin 'http://example.com/private/page'

(This is one line, though it likely wraps on your browser)

If you want the output saved to a file, change -O - to -O /some/file/name.ext

You don't need cURL to do POSTed form data. --post-data 'key1=value1&key2=value2' works just fine. Note: you can also pass a file name to wget with the POST data in the file.

A solution which uses lynx and wget.

Note: Lynx has to have been compiled with the --enable-persistent-cookies flag for this to work

When you want to use wget to download some file from a site which requires login, you just need a cookie file. In order to generate the cookie file, I choose lynx. lynx is a text web browser. First you need a configure file for lynx to save cookie. Create a file lynx.cfg. Write these configuration into the file.

SET_COOKIES:TRUE
ACCEPT_ALL_COOKIES:TRUE
PERSISTENT_COOKIES:TRUE
COOKIE_FILE:cookie.file

Then start lynx with this command:

lynx -cfg=lynx.cfg http://the.site.com/login

After you input the username and password, and select 'preserve me on this pc' or something similar. If login successfully, you will see a beautiful text web page of the site. And you logout. The in the current directory, you will find a cookie file named as cookie.file. This is what we need for wget.

Then wget can download file from the site with this command.

wget --load-cookies ./cookie.file http://the.site.com/download/we-can-make-this-world-better.tar.gz

I use this chrome extension. It'll give you the wget command for any download link you open.

Related