jsoup posting and cookie

user_809391 0
Viewed 63376

I'm trying to use jsoup to login to a site and then scrape information, I am running into in a problem, I can login successfully and create a Document from index.php but I cannot get other pages on the site. I know I need to set a cookie after I post and then load it when I'm trying to open another page on the site. But how do I do this? The following code lets me login and get index.php

Document doc = Jsoup.connect("http://www.example.com/login.php")
               .data("username", "myUsername", 
                     "password", "myPassword")
               .post();

I know I can use apache httpclient to do this but I don't want to.

jsoup screen-scraping java
6 Answers
Connection.Response res = Jsoup.connect("http://www.example.com/login.php")
    .data("username", "myUsername")
    .data("password", "myPassword")
    .method(Connection.Method.POST)
    .execute();
//Connecting to the server with login details
Document doc = res.parse();
//This will give the redirected file
Map<String,String> cooki=res.cookies();
//This gives the cookies stored into cooki
Document docs= Jsoup.connect("http://www.example.com/otherPage")
    .cookies(cooki)
    .get();
//This gives the data of the required website
user_136874630

Why reconnect? if there are any cookies to avoid 403 Status i do so.

                Document doc = null;
                int statusCode = -1;
                String statusMessage = null;
                String strHTML = null;
        
                try {
    // connect one time.                
                    Connection con = Jsoup.connect(urlString);
    // get response.
                    Connection.Response res = con.execute();        
    // get cookies
                    Map<String, String> loginCookies = res.cookies();

    // print cookie content and status message
                    if (loginCookies != null) {
                        for (Map.Entry<String, String> entry : loginCookies.entrySet()) {
                            System.out.println(entry.getKey() + ":" + entry.getValue().toString() + "\n");
                        }
                    }
        
                    statusCode = res.statusCode();
                    statusMessage = res.statusMessage();
                    System.out.print("Status CODE\n" + statusCode + "\n\n");
                    System.out.print("Status Message\n" + statusMessage + "\n\n");
        
    // set login cookies to connection here
                    con.cookies(loginCookies).userAgent("Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) Gecko/20100101 Firefox/23.0");
        
    // now do whatever you want, get document for example
                    doc = con.get();
    // get HTML
                    strHTML = doc.head().html();

                } catch (org.jsoup.HttpStatusException hse) {
                    hse.printStackTrace();
                } catch (IOException ioe) {
                    ioe.printStackTrace();
                }
user_149696060
Related