I am trying to write something which will continuously update a csv file.
There can be some overlaps since my script collects a little more data than is required so I want to use anti_join to find the data points which are not in the current saved .csv file. I want to append the new data points to the top of the .csv file.
Currently what I have is something like:
iris = iris %>%
mutate(rowId = row_number())
df1 = iris[1:15, ]
df2 = iris[1:10, ]
df1 %>%
anti_join(df2, by = "rowId")
#write.csv(., append = TRUE)
Which gives me correctly the missing data (by rowId)
Sepal.Length Sepal.Width Petal.Length Petal.Width Species rowId
1 5.4 3.7 1.5 0.2 setosa 11
2 4.8 3.4 1.6 0.2 setosa 12
3 4.8 3.0 1.4 0.1 setosa 13
4 4.3 3.0 1.1 0.1 setosa 14
5 5.8 4.0 1.2 0.2 setosa 15
I want to append this top the original data df1 using write_csv. However, my current method adds the data to the bottom of the .csv file. I could sort the data but when the data set gets really large, sorting the data each time just to append correctly 5 observations at the top of the .csv file is not computationally feasible (i.e. when there are 1,000,000 observations, to sort all 1,000,005 observations to have the 5 at the top)
How can I use dply / tidyverse to append the rows to the top of the data?