nothing to commit, working tree clean Git Hub Action and R script

Viewed 43

I'm trying to run this simple scraping script with Git Hub Action:

It simply takes a vector of ASIN, generates the URL for the Amazon product page and extract the name from the given xpath, then it generates a dataframe and export it as a .csv file.

I would like to run this script every hour and output a different .csv file

For the future I would like to run the script every 24h and within the action use the GMAIL api to send it via mail.

For now I'm just trying to figure out how to obtain the basic task

library(tidyverse)
library(tibble)
library(openxlsx)
library(gdata)
library(lubridate)
library(rvest)
library(stringr)
library(dplyr)
library(purrr)
library(plyr)


ua <-
  "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"




name_checker_2 <- function(asin) {
  url_compositor <- paste0("https://www.amazon.it/dp/", asin)
  
  system('sleep 15')
  
  page <- read_html(url_compositor, user_agent = ua)
  
  name <- page %>%
    html_nodes(xpath = "//*[@id='productTitle']/text()") %>%
    html_text()
  
  name <- trimws(name)
  
  asin_2 <- unlist(asin)
  
  final <- cbind(asin, asin_2, name)
  
  final_df <<- as.data.frame(final)
  
  return(final_df)
  
  print("DONE")
  
}



asin <- c(
  "B07JVLQ38M",
  "B08LVXG2V1",
  "B07Y8DPHCL",
  "B082WSYYCX",
  "B082WSZXHB",
  "B09CTY5P2L",
  "B09SGB55N4",
  "B015O6A6GM",
  "B00G3DNXIE",
  "B07Y8DHN3F",
  "B015O6AFPE",
  "B00O4QVS66",
  "B093C19CSD",
  "B079Z9GXQS",
  "B09ZJ9QXZY",

)

date <- Sys.time()

scrape_raw <- purrr::map(asin, purrr::possibly(name_checker_2, NA))

product_name_df_delay_ua <- plyr::ldply(scrape_raw, data.frame)

write.csv(product_name_df_delay_ua, file = paste0("data/Amazon_name_",date,".csv"))

And this is my YAML file

# Hourly scraping
name: amazonR

# Controls when the action will run.
on:
  schedule:
    - cron: '0 * * * *'

jobs:
  autoscrape:
    # The type of runner that the job will run on
    runs-on: macos-latest

    # Load repo and install R
    steps:
    - uses: actions/checkout@master
    - uses: r-lib/actions/setup-r@master

    # Set-up R
    - name: Install packages
      run: |
        R -e 'install.packages("tidyverse")'
        R -e 'install.packages("tibble")'
        R -e 'install.packages("openxlsx")'
        R -e 'install.packages("gdata")'
        R -e 'install.packages("lubridate")'
        R -e 'install.packages("rvest")'
        R -e 'install.packages("stringr")'
        R -e 'install.packages("dplyr")'
        R -e 'install.packages("purrr")'
        R -e 'install.packages("plyr")'
            
    # Run R script
    - name: Scrape
      run: Rscript amazon_name_checker_simplest.R

    # Add new files in data folder, commit along with other modified files, push
    - name: Commit files
      run: |
        git config --local user.name github-actions
        git config --local user.email "actions@github.com"
        git add data/*
        git commit -am "GH ACTION Autorun $(date)"
        git push origin main
      env:
        REPO_KEY: ${{secrets.GITHUB_TOKEN}}
        username: github-actions

The action run correctly the first time but the last run it gave me this error:

Run git config --local user.name github-actions
  git config --local user.name github-actions
  git config --local user.email "actions@github.com"
  git add data/*
  git commit -am "GH ACTION Autorun $(date)"
  git push origin main
  shell: /bin/bash -e {0}
  env:
    R_LIBS_USER: /Users/runner/work/_temp/Library
    TZ: UTC
    _R_CHECK_SYSTEM_CLOCK_: FALSE
    NOT_CRAN: true
    REPO_KEY: ***
    username: github-actions
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean
Error: Process completed with exit code 1.
1 Answers

I fixed the problem. I wrongly assumed that the output file would be overwritten since the file name was the same but it did not. And for what I saw since no new file was made no new commit as consequence.

I fixed it giving a dynamic name to the output file, including the sys.time() in the naming

This is the new version of the write.csv function:

date <- Sys.time()

scrape_raw <- purrr::map(asin, purrr::possibly(name_checker_2, NA))

product_name_df_delay_ua <- plyr::ldply(scrape_raw, data.frame)

write.csv(product_name_df_delay_ua, file = paste0("data/Amazon_name_",date,".csv"))

I hope it will be useful to others with the same problem.

Related