How to load lazy content on Linkedin search page using selenium

I have checked the div structure and the way linkedin is showing the results. So, if you hit the url directly and check the by following xpath: //li[contains(@class,'search-result')] You would find out that all the results are already loaded on the page, but linkedin are showing only 5 results in one go and on scrolling, it shows the next 5 results, however all the results are already loaded on the page and can be found out by the mentioned xpath.

Refer to this image which highlights the div structure and results when you find the results on entering the xpath on hitting the url: https://imgur.com/Owu4NPh and
Refer to this image which highlights the div structure and results after scrolling the page to the bottom and then finding the results using the same xpath: https://imgur.com/7WNR830

You could see the result set is same however there is an additional search-result__occlusion-hint part in the < li > tag in the last 5 results and through this linkedin is hiding the next 5 results and showing only the first 5 results on the first go.

Now comes the implementation part, i have checked "Next" button comes only when you scroll through whole results on the page, so instead of scrolling to a definite coordinates because that can be changed for different screensizes and windows, you can take the results in a list of webelement and get it's size and then scroll to the last element of that list. In this case, if there are total 10 results then the page will be scrolled to the 10th results and if there are only 4 results then the page will be scrolled to the 4th result and after scrolling you can check if the Next button is present on the page or not. For this, you can check the list size of the "Next" button web element list, if the list size is greater than 0, it means the next button is present on the page and if its not greater than 0, that means the Next button is not present on the list and you can stop your execution there.

So to implement it, i have taken a boolean which has an initial value as true and the code will be run in a loop till that boolean becomes false and it will become false when the Next button list size becomes equal to 0.

Please refer to the code below:

public class App 
{    
    WebDriver driver;

  // For initialising javascript executor
  public Object executeScript(String script, Object... args) {
    JavascriptExecutor exe = (JavascriptExecutor) driver;
    return exe.executeScript(script, args);
  }

  // Method for scrolling to the element
  public void scrollToElement(WebElement element) {
    executeScript("window.scrollTo(arguments[0],arguments[1])", element.getLocation().x, element.getLocation().y);

    }

  public static void main(String[] args) {
    // You can change the driver to bot according to your usecase
    driver = new FirefoxDriver();
    // Add your direct URL here and perform the login after that, if necessary
    driver.get(url);
    // Wait for the URL to load completely
    Thread.sleep(10000);
    // Initialising the boolean
    boolean nextButtonPresent = true;
    while (nextButtonPresent) {
        // Fetching the results on the page by the xpath
        List<WebElement> results = driver.findElements(By.xpath("//li[contains(@class,'search-result')]"));
        // Scrolling to the last element in the list
        scrollToElement(results.get(results.size() - 1));
        Thread.sleep(2000);

        // Checking if next button is present on the page
        List<WebElement> nextButton = driver.findElements(By.xpath("//button[@class='next']"));
        if (nextButton.size() > 0) {
            // If yes then clicking on it
            nextButton.get(0).click();
            Thread.sleep(10000);
        } else {
            // Else setting the boolean as false
            nextButtonPresent = false;
            System.out.println("Next button is not present, so ending the script");
        }
      }
   }
}

What I have observed was, the content is already loaded in the page and it will be displayed to us when we do the scroll down.

But if we inspect the 'Next >' button by loading the page manually by using the class name 'next' for example like below,

//button[@class='next']

we cannot locate it until we do the scroll down because it is not visible to us. But by using the below XPath, we can identify all the profile links count irrespective of whether they are displayed or not?

//h3[contains(@class, 'search-results__total')]/parent::div/ul/li

As you want to fetch all the profile links from the page, we can use the above XPath help to do that. We will get the links count using the above XPath then we will scroll into each element view at a time and then we will fetch the profile links on the way like below :

// Identifying the all the profile links
List<WebElement> totalProfileLinks = driver.findElements(By.xpath("//h3[contains(@class, 'search-results__total')]/parent::div/ul/li"));
// Looping for getting the profile link
for(int i=1;i<totalProfileLinks.size();i++) {
    // Scrolling so that it will be visible
    ((JavascriptExecutor) driver).executeScript("arguments[0].scrollIntoView(true);", totalProfileLinks.get(i));
    // Fetching the anchor node
    final WebElement link = driver.findElement(By.xpath("(//h3[contains(@class, 'search-results__total')]/parent::div/ul/li//div[contains(@class, 'search-result__info')]//a)["+i+"]"));
    // Avoiding the StaleElementReferenceException
    new FluentWait<WebDriver>(driver).withTimeout(1, TimeUnit.MINUTES).pollingEvery(1, TimeUnit.SECONDS).ignoring(StaleElementReferenceException.class).until(new Function<WebDriver, WebElement>() {
        public WebElement apply(WebDriver arg0) {
            return link;
        }
    });
    // Fetching and printing the link from anchor node
    System.out.println(link.getAttribute("href").trim());
}

So, if we want to click on the 'Next >' button first we need to check if its present or not(As we have scrolled while fetching the profile links, the 'next' button also get displyed). We can use the help of `driver.findElements();` method to get the matches of that element count and can store it in some List(Because it returns List of WebElements) like below :

List<WebElement> nextButton = driver.findElements(By.className("next"));

The benefit of using the above technique is, the script won't fail if there are no element matches also and we will have an empty list if there are no matches.

Then we can use the size() method of the List interface to get the matches count like below :

int size = nextButton.size();

And if the size is more than 0 then that element is present otherwise not, we can check that condition like below :

if(size > 0) {
    nextButton.get(0).click(); // Do some operation like clicking on it
    System.out.println("=> 'Next >' button is there and clicked on it...");
} else {
    System.out.println("=> 'Next >' button is NOT there...");
}

As the content is loaded and the element is view-able, we will use the JavaScriptExecutor which will locate and clicks on it.

Wrap the above code in while loop and check the presence of the 'Next >' button every time after clicking on the previous 'Next >' button like below :

boolean next = true;
while(next) {
    // Checking 'Next >' button is there or not in the page
    List<WebElement> nextButton = driver.findElements(By.className("next"));
    // If the 'Next >' button is there then clicking on it otherwise stopping the execution
    if(nextButton.size() > 0) {
        doClickUsingJSE(nextButton.get(0));
        System.out.println("=> 'Next >' button is there and clicked on it...");
    } else {
        next = false;
        System.out.println("=> 'Next >' button is NOT there so stopping the execution...");
    }
    Thread.sleep(1000);
}

Loop will break if the 'if' condition fails in the above code because 'next' will becomes 'false'. And if we use the Fluent Wait then it will help us in avoiding some 'Exceptions' like 'WebDriverException' and 'StaleElementReferenceException'. So I have written one separate method which will wait for an element by avoiding some exceptions and clicks on it if the conditions get satisfied.

Check the code below :

private static void doClickUsingJSE(final WebElement element) {
    // Using the Fluent Wait to avoid some exceptions like WebDriverException and StaleElementReferenceException
    Wait<WebDriver> wait = new FluentWait<WebDriver>(driver).withTimeout(1, TimeUnit.MINUTES).pollingEvery(1, TimeUnit.SECONDS).ignoring(WebDriverException.class, StaleElementReferenceException.class);
    WebElement waitedElement = wait.until(new Function<WebDriver, WebElement>() {
        public WebElement apply(WebDriver driver) {
            return element;
        }
    });
    wait.until(ExpectedConditions.visibilityOf(waitedElement));
    wait.until(ExpectedConditions.elementToBeClickable(waitedElement));
    // Clicking on the particular element using the JavaScriptExcecutor
    ((JavascriptExecutor) driver).executeScript("arguments[0].click();", waitedElement);
    }

As I mentioned about the JavaScriptExecutor earlier, I have included use of that also in the above method only.

Try the below end to end working code :

import java.util.List;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.By;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.Keys;
import org.openqa.selenium.StaleElementReferenceException;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebDriverException;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.FluentWait;
import org.openqa.selenium.support.ui.Wait;

import com.google.common.base.Function;

public class BasePage 
{
    // Declaring WebDriver
    private static WebDriver driver;

    private static void doClickUsingJSE(final WebElement element) {
        // Using the Fluent Wait to avoid some exceptions like WebDriverException and StaleElementReferenceException
        Wait<WebDriver> wait = new FluentWait<WebDriver>(driver).withTimeout(1, TimeUnit.MINUTES).pollingEvery(1, TimeUnit.SECONDS).ignoring(WebDriverException.class, StaleElementReferenceException.class);
        WebElement waitedElement = wait.until(new Function<WebDriver, WebElement>() {
            public WebElement apply(WebDriver driver) {
                return element;
            }
        });
        wait.until(ExpectedConditions.visibilityOf(waitedElement));
        wait.until(ExpectedConditions.elementToBeClickable(waitedElement));
        // Clicking on the particular element using the JavaScriptExcecutor
        ((JavascriptExecutor) driver).executeScript("arguments[0].click();", waitedElement);
    }

    public static void main( String[] args ) throws Exception
    {
        System.setProperty("webdriver.chrome.driver", "C:\\NotBackedUp\\chromedriver.exe");

        // Initializing the Chrome Driver
        driver = new ChromeDriver();

        // Launching the LinkedIn site
        driver.get("https://linkedin.com/search/results/people/?facetGeoRegion=[\"tr%3A0\"]&facetNetwork=[\"F\"]&origin=FACETED_SEARCH&page=YOUR_PAGE_NUMBER");

        // You can avoid this and it to your convience way
        // As there are no connections in my page, I have used like this
        //------------------------------------------------------------------------------------
        // Switching to the login from - iframe involved
        driver.switchTo().frame(driver.findElement(By.className("authentication-iframe")));

        // Clicking on the Sign In button
        doClickUsingJSE(driver.findElement(By.xpath("//a[text()='Sign in']")));

        // Entering the User Name
        WebElement element = driver.findElement(By.id("username"));
        doClickUsingJSE(element);
        element.sendKeys("something@gmail.com");

        // Entering the Password
        element = driver.findElement(By.id("password"));
        doClickUsingJSE(element);
        element.sendKeys("anything"+Keys.ENTER);

        // Clicking on the People drop down
        Thread.sleep(8000);
        element = driver.findElement(By.xpath("//span[text()='People']"));
        doClickUsingJSE(element);

        // Selecting the All option
        Thread.sleep(2000);
        element = driver.findElement(By.xpath("//ul[@class='list-style-none']/li[1]"));
        element.click();

        // Searching something in the LinkedIn search box
        Thread.sleep(3000);
        element = driver.findElement(By.xpath("//input[@role='combobox']"));
        doClickUsingJSE(element);
        element.sendKeys("a"+Keys.ENTER);
        Thread.sleep(8000);
        //------------------------------------------------------------------------------------

        boolean next = true;
        while(next) {
            // Identifying the all the profile links
            List<WebElement> totalProfileLinks = driver.findElements(By.xpath("//h3[contains(@class, 'search-results__total')]/parent::div/ul/li"));
            // Looping for getting the profile link
            for(int i=1;i<totalProfileLinks.size();i++) {
                // Scrolling so that it will be visible
                ((JavascriptExecutor) driver).executeScript("arguments[0].scrollIntoView(true);", totalProfileLinks.get(i));
                // Fetching the anchor node
                final WebElement link = driver.findElement(By.xpath("(//h3[contains(@class, 'search-results__total')]/parent::div/ul/li//div[contains(@class, 'search-result__info')]//a)["+i+"]"));
                // Avoiding the StaleElementReferenceException
                new FluentWait<WebDriver>(driver).withTimeout(1, TimeUnit.MINUTES).pollingEvery(1, TimeUnit.SECONDS).ignoring(StaleElementReferenceException.class).until(new Function<WebDriver, WebElement>() {
                    public WebElement apply(WebDriver arg0) {
                        return link;
                    }
                });
                // Fetching and printing the link from anchor node
                System.out.println(link.getAttribute("href").trim());
            }

            // Checking 'Next >' button is there or not in the page
            List<WebElement> nextButton = driver.findElements(By.className("next"));
            // If the 'Next >' button is there then clicking on it otherwise stopping the execution
            if(nextButton.size() > 0) {
                doClickUsingJSE(nextButton.get(0));
                System.out.println("=> 'Next >' button is there and clicked on it...");
            } else {
                next = false;
                System.out.println("=> 'Next >' button is NOT there so stopping the execution...");
            }
            Thread.sleep(1000);
        }

    }
}

I hope it helps... Happy Coding...