Trying to loop through a webpage to scrape all football players names, but only getting the first one?

You are using a driver method which only returns a single value find_element_by_class_name, switch to find_elements_by_class_name to get a list then iterate through the list:

names = driver.find_elements_by_class_name('sidearm-roster-player-name')
for name in names:
    player_name = name.find_element_by_tag_name('a').text
    print(player_name)

In my case at least an User-Agent header was required and I could then use requests. You can then gather the parent nodes with a css class selector and then loop those parents and extract required info into a dataframe; again, using faster and short css selectors. As mentioned, the key is to gather all parent nodes in this case by using select. This has less overhead than selenium.

Py:

from bs4 import BeautifulSoup as bs
import requests
import pandas as pd

r = requests.get('https://rolltide.com/roster.aspx?roster=226&path=football', headers = {'User-Agent':'Mozilla/5.0'})
soup = bs(r.content, 'lxml')
results = {}

for num, p in enumerate(soup.select('.sidearm-roster-player')):
    results[num] = {'position': p.select_one('.sidearm-roster-player-position >span:first-child').text.strip()
           ,'height': p.select_one('.sidearm-roster-player-height').text
           ,'weight': p.select_one('.sidearm-roster-player-weight').text
           ,'number': p.select_one('.sidearm-roster-player-jersey-number').text
           ,'name': p.select_one('.sidearm-roster-player-name a').text
           ,'year': p.select_one('.sidearm-roster-player-academic-year').text
           ,'hometown': p.select_one('.sidearm-roster-player-hometown').text
           ,'highschool': p.select_one('.sidearm-roster-player-highschool').text
          }
df = pd.DataFrame(results.values(), columns = ['position','height','weight','number','name','year','hometown','highschool'])
print(df)

R:

purrr is used to handle loop over parent nodes to write to df. str_squish from stringr is used to tidy up output of one child node in loop. httr is used to provide header.

library(httr)
library(purrr)
library(rvest)
library(stringr)

headers = c('User-Agent' = 'Mozilla/5.0')
pg <- content(httr::GET(url = 'https://rolltide.com/roster.aspx?roster=226&path=football', httr::add_headers(.headers=headers)))

df <- map_df(pg%>%html_nodes('.sidearm-roster-player'), function(item) {

     data.frame(position = str_squish(item%>%html_node('.sidearm-roster-player-position >span:first-child')%>%html_text()),
                height = item%>%html_node('.sidearm-roster-player-height')%>%html_text(),
                weight = item%>%html_node('.sidearm-roster-player-weight')%>%html_text(),
                number = item%>%html_node('.sidearm-roster-player-jersey-number')%>%html_text(),
                name = item%>%html_node('.sidearm-roster-player-name a')%>%html_text(),
                year = item%>%html_node('.sidearm-roster-player-academic-year')%>%html_text(),
                hometown = item%>%html_node('.sidearm-roster-player-hometown')%>%html_text(),
                highschool = item%>%html_node('.sidearm-roster-player-highschool')%>%html_text(),
                stringsAsFactors=FALSE)
     })

View(df)

For anyone wanting to use R (rvest), here is the code that collects the roster data into a data frame:

library(tidyverse)
library(magrittr)
library(rvest)

url <- "https://rolltide.com/roster.aspx?roster=226&path=football"
page <- url %>% read_html()

position <- list()
height <- list()
weight <- list()
number <- list()
name <- list()
yr <- list()
hometown <- list()
high.school <- list()

for (i in seq(1,250)) {
    position[[i]] <- page %>% html_nodes(xpath=paste0('//*[@id="main-content"]/article/div[4]/div/div[1]/div[2]/div[1]/section/ul/li[',i,']/div[1]/div[1]/div[2]/div[1]/span[1]/text()')) %>% xml_text %>% str_trim
    height[[i]] <- page %>% html_nodes(xpath=paste0('//*[@id="main-content"]/article/div[4]/div/div[1]/div[2]/div[1]/section/ul/li[',i,']/div[1]/div[1]/div[2]/div[1]/span[2]')) %>% xml_text
    weight[[i]] <- page %>% html_nodes(xpath=paste0('//*[@id="main-content"]/article/div[4]/div/div[1]/div[2]/div[1]/section/ul/li[',i,']/div[1]/div[1]/div[2]/div[1]/span[3]/text()')) %>% xml_text
    number[[i]] <- page %>% html_nodes(xpath=paste0('//*[@id="main-content"]/article/div[4]/div/div[1]/div[2]/div[1]/section/ul/li[',i,']/div[1]/div[1]/div[2]/div[2]/span/span')) %>% xml_text
    name[[i]] <- page %>% html_nodes(xpath=paste0('//*[@id="main-content"]/article/div[4]/div/div[1]/div[2]/div[1]/section/ul/li[',i,']/div[1]/div[1]/div[2]/div[2]/p/a')) %>% xml_text
    yr[[i]] <- page %>% html_nodes(xpath=paste0('//*[@id="main-content"]/article/div[4]/div/div[1]/div[2]/div[1]/section/ul/li[',i,']/div[1]/div[2]/div[1]/span[1]')) %>% xml_text
    hometown[[i]] <- page %>% html_nodes(xpath=paste0('//*[@id="main-content"]/article/div[4]/div/div[1]/div[2]/div[1]/section/ul/li[',i,']/div[1]/div[2]/div[1]/span[2]/text()')) %>% xml_text
    high.school[[i]] <- page %>% html_nodes(xpath=paste0('//*[@id="main-content"]/article/div[4]/div/div[1]/div[2]/div[1]/section/ul/li[',i,']/div[1]/div[2]/div[1]/span[3]/text()')) %>% xml_text
}

position    %<>% tibble %>% unnest
height      %<>% tibble %>% unnest
weight      %<>% tibble %>% unnest
number      %<>% tibble %>% unnest
name        %<>% tibble %>% unnest
yr          %<>% tibble %>% unnest
hometown    %<>% tibble %>% unnest
high.school %<>% tibble %>% unnest

final <- bind_cols(position,height,weight,number,name,yr,hometown,high.school)
names(final) <- c("position","height","weight","number","name","yr","hometown","high.school")

The trick is to select the Xpath instead of the CSS selector and use xpath= in the html_nodes() call.

It's obviously a bit uglier, but it doesn't require Selenium or other heavy setups.

EDIT: You should look at QHarr's answer above for more streamlined code.