I wrote a script in R that reads in one excel file and then creates/renames variables, changes variable type, fills in values, as well as filters rows as necessary in order to clean up the file. My question is how do I apply that script to all excel files in a specific folder and then set them together once they've all been cleaned up.
The original script is:
library(xlsx)
install.packages("tidyverse")
library(tidyverse)
input_data_directory = '/Users/A/Documents/folder1'
# UPDATE BEFORE RUNNING CODE
input_file = '/wasde0822.xls'
# PROCESSING THE DATA
# reading in source file
wasde1 <- read.xlsx(paste0(input_data_directory, input_file),
sheetName = "Page 8",
colIndex = 1:8,
rowIndex = 9:33)
Date <- read.xlsx(paste0(input_data_directory, input_file),
sheetName = "Page 8",
header = F,
startRow = 1,
endRow = 1,
colIndex = 1)
# renaming fields
names(wasde1)[names(wasde1) == 'World'] <- "Product"
names(wasde1)[names(wasde1) == 'NA.'] <- "Time"
names(wasde1)[names(wasde1) == 'NA..1'] <- "Month"
names(wasde1)[names(wasde1) == 'Total.Supply'] <- "TotalSupply"
names(wasde1)[names(wasde1) == 'Trade.2.'] <- "Trade"
names(wasde1)[names(wasde1) == 'Total.Use.3.'] <- "TotalUse"
names(wasde1)[names(wasde1) == 'Ending.Stocks'] <- "EndingStocks"
# filtering out unnecessary rows
wasde1 = wasde1[-1,]
wasde1 <- wasde1[!is.na(wasde1$Time),]
# filling in values
wasde1 <- wasde1 %>% fill(Product, .direction = "down")
# creating new fields
wasde1$Geography <- "World"
wasde1$Date <- paste(Date)
# changing character fields to numeric fields
wasde1$Output <- as.numeric(wasde1$Output)
wasde1$TotalSupply <- as.numeric(wasde1$TotalSupply)
wasde1$Trade <- as.numeric(wasde1$Trade)
wasde1$TotalUse <- as.numeric(wasde1$TotalUse)
wasde1$EndingStocks <- as.numeric(wasde1$EndingStocks)
My attempts to put this in a loop to apply to more than one excel file were not successful. I'm able to read multiple excel files in a loop (below code, which works), but the rest has eluded me; any help would be appreciated. I'm sure this is possible; it's my R knowledge that is lacking.
library(xlsx)
install.packages("tidyverse")
library(tidyverse)
input_data_directory = '/Users/A/Documents/folder1'
filename_list <- list.files(input_data_directory,pattern="*.xls", full.names = FALSE)
for (i in 1:length(filename_list)) {
# reading in source file
assign(filename_list[i], read.xlsx(paste0(input_data_directory,filename_list[i]),
sheetName = "Page 8",
colIndex = 1:8,
rowIndex = 9:33))
assign(paste("date",i,sep=""), read.xlsx(paste0(input_data_directory, filename_list[i]),
sheetName = "Page 8",
header = F,
startRow = 1,
endRow = 1,
colIndex = 1))
}
Apologies in advance for not providing sample data. The best I'm able to do is provide a link to one sample file. This link will take you to a U.S. Department of Agriculture page. On that page, click on the excel report link circled below to get a sample file:
