How do i loop through divs using jsoup

Question

Hi guys I'm using jsoup in a java webapplication on IntelliJ. I'm trying to scrape data of port call events from a shiptracking website and store the data in a mySQL database.

The data for the events is organised in divs with the class name table-group and the values are in another div with the class name table-row.
My problem is the divs rows for all the vessel are all the same class name and im trying to loop through each row and push the data to a database. So far i have managed to create a java class to scrape the first row.
How can i loop through each row and store those values to my database. Should i create an array list to store the values?

this is my scraper class

public class Scarper {

    private static Document doc;

    public static void main(String[] args) {

        final String url =
                "https://www.myshiptracking.com/ports-arrivals-departures/?mmsi=&pid=277&type=0&time=&pp=20";

        try {

            doc = Jsoup.connect(url).get();
        } catch (IOException e) {
            e.printStackTrace();
        }
        Events();
    }

    public static void Events() {
        Elements elm = doc.select("div.table-group:nth-of-type(2) > .table-row");

        List arrayList = new ArrayList();

        for (Element ele : elm) {

            String event = ele.select("div.col:nth-of-type(2)").text();
            String time = ele.select("div.col:nth-of-type(3)").text();
            String port = ele.select("div.col:nth-of-type(4)").text();
            String vessel = ele.select(".td_vesseltype.col").text();
            Event ev = new Event();
            System.out.println(event);
            System.out.println(time);
            System.out.println(port);
            System.out.println(vessel);
        }
    }
}

sample of the div classes i want to scrape


            
                
                    
                    Event
                    Time (MT)
                    Port
                    Vessel
                
                                    
                    
                        
                        Departure
                        2022-02-14 16:51
                        BELFAST
                        WILSON BLYTH [GB]
                    
                
                                    
                    
                        
                        Arrival
                        2022-02-14 16:51
                        HUNTERS QUAY
                        SOUND OF SOAY [GB]
                    
                
                                    
                    
                        
                        Departure
                        2022-02-14 16:51
                        LARGS
                        LOCH SHIRA [GB]
                    
                
                                    
                    
                        
                        Departure
                        2022-02-14 16:51
                        RYDE
                        ISLAND FLYER [GB]

user_3426328 · Answer

You can start with looping over the table's rows: the selector for the table is .cs-table so you can get the table with Element table = doc.select(".cs-table").first();. Next you can get the table's rows with the selector div.table-row - Elements rows = doc.select("div.table-row"); now you can loop over all the rows and extract the data from each row. The code should look like:

Element table = doc.select(".cs-table").first();
Elements rows = doc.select("div.table-row");
for (Element row : rows) {
        String event = row.select("div.col:nth-of-type(2)").text();
        String time = row.select("div.col:nth-of-type(3)").text();
        String port = row.select("div.col:nth-of-type(4)").text();
        String vessel = row.select(".td_vesseltype.col").text();
        System.out.println(event + "-" + time + " " + port + " " + vessel);
        System.out.println("---------------------------");
        // Do stuff with data here
    }

Now it's up to you to decide if you want to keep the data in some array/list inside the loop and use it later, or to insert it directly to your database.

How do i loop through divs using jsoup

1 Answers