I'm trying to run this simple scraping script with Git Hub Action:
It simply takes a vector of ASIN, generates the URL for the Amazon product page and extract the name from the given xpath, then it generates a dataframe and export it as a .csv file.
I would like to run this script every hour and output a different .csv file
For the future I would like to run the script every 24h and within the action use the GMAIL api to send it via mail.
For now I'm just trying to figure out how to obtain the basic task
library(tidyverse)
library(tibble)
library(openxlsx)
library(gdata)
library(lubridate)
library(rvest)
library(stringr)
library(dplyr)
library(purrr)
library(plyr)
ua <-
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"
name_checker_2 <- function(asin) {
url_compositor <- paste0("https://www.amazon.it/dp/", asin)
system('sleep 15')
page <- read_html(url_compositor, user_agent = ua)
name <- page %>%
html_nodes(xpath = "//*[@id='productTitle']/text()") %>%
html_text()
name <- trimws(name)
asin_2 <- unlist(asin)
final <- cbind(asin, asin_2, name)
final_df <<- as.data.frame(final)
return(final_df)
print("DONE")
}
asin <- c(
"B07JVLQ38M",
"B08LVXG2V1",
"B07Y8DPHCL",
"B082WSYYCX",
"B082WSZXHB",
"B09CTY5P2L",
"B09SGB55N4",
"B015O6A6GM",
"B00G3DNXIE",
"B07Y8DHN3F",
"B015O6AFPE",
"B00O4QVS66",
"B093C19CSD",
"B079Z9GXQS",
"B09ZJ9QXZY",
)
date <- Sys.time()
scrape_raw <- purrr::map(asin, purrr::possibly(name_checker_2, NA))
product_name_df_delay_ua <- plyr::ldply(scrape_raw, data.frame)
write.csv(product_name_df_delay_ua, file = paste0("data/Amazon_name_",date,".csv"))
And this is my YAML file
# Hourly scraping
name: amazonR
# Controls when the action will run.
on:
schedule:
- cron: '0 * * * *'
jobs:
autoscrape:
# The type of runner that the job will run on
runs-on: macos-latest
# Load repo and install R
steps:
- uses: actions/checkout@master
- uses: r-lib/actions/setup-r@master
# Set-up R
- name: Install packages
run: |
R -e 'install.packages("tidyverse")'
R -e 'install.packages("tibble")'
R -e 'install.packages("openxlsx")'
R -e 'install.packages("gdata")'
R -e 'install.packages("lubridate")'
R -e 'install.packages("rvest")'
R -e 'install.packages("stringr")'
R -e 'install.packages("dplyr")'
R -e 'install.packages("purrr")'
R -e 'install.packages("plyr")'
# Run R script
- name: Scrape
run: Rscript amazon_name_checker_simplest.R
# Add new files in data folder, commit along with other modified files, push
- name: Commit files
run: |
git config --local user.name github-actions
git config --local user.email "actions@github.com"
git add data/*
git commit -am "GH ACTION Autorun $(date)"
git push origin main
env:
REPO_KEY: ${{secrets.GITHUB_TOKEN}}
username: github-actions
The action run correctly the first time but the last run it gave me this error:
Run git config --local user.name github-actions
git config --local user.name github-actions
git config --local user.email "actions@github.com"
git add data/*
git commit -am "GH ACTION Autorun $(date)"
git push origin main
shell: /bin/bash -e {0}
env:
R_LIBS_USER: /Users/runner/work/_temp/Library
TZ: UTC
_R_CHECK_SYSTEM_CLOCK_: FALSE
NOT_CRAN: true
REPO_KEY: ***
username: github-actions
On branch main
Your branch is up to date with 'origin/main'.
nothing to commit, working tree clean
Error: Process completed with exit code 1.