Workflow for statistical analysis and report writing

user_2554948 0
Viewed 43729

Does anyone have any wisdom on workflows for data analysis related to custom report writing? The use-case is basically this:

  1. Client commissions a report that uses data analysis, e.g. a population estimate and related maps for a water district.

  2. The analyst downloads some data, munges the data and saves the result (e.g. adding a column for population per unit, or subsetting the data based on district boundaries).

  3. The analyst analyzes the data created in (2), gets close to her goal, but sees that needs more data and so goes back to (1).

  4. Rinse repeat until the tables and graphics meet QA/QC and satisfy the client.

  5. Write report incorporating tables and graphics.

  6. Next year, the happy client comes back and wants an update. This should be as simple as updating the upstream data by a new download (e.g. get the building permits from the last year), and pressing a "RECALCULATE" button, unless specifications change.

At the moment, I just start a directory and ad-hoc it the best I can. I would like a more systematic approach, so I am hoping someone has figured this out... I use a mix of spreadsheets, SQL, ARCGIS, R, and Unix tools.

Thanks!

PS:

Below is a basic Makefile that checks for dependencies on various intermediate datasets (w/ .RData suffix) and scripts (.R suffix). Make uses timestamps to check dependencies, so if you touch ss07por.csv, it will see that this file is newer than all the files / targets that depend on it, and execute the given scripts in order to update them accordingly. This is still a work in progress, including a step for putting into SQL database, and a step for a templating language like sweave. Note that Make relies on tabs in its syntax, so read the manual before cutting and pasting. Enjoy and give feedback!

http://www.gnu.org/software/make/manual/html_node/index.html#Top

R=/home/wsprague/R-2.9.2/bin/R

persondata.RData : ImportData.R ../../DATA/ss07por.csv Functions.R
   $R --slave -f ImportData.R

persondata.Munged.RData : MungeData.R persondata.RData Functions.R
      $R --slave -f MungeData.R

report.txt:  TabulateAndGraph.R persondata.Munged.RData Functions.R
      $R --slave -f TabulateAndGraph.R > report.txt
data-visualization statistics r
14 Answers

For creating custom reports, I've found it useful to incorporate many of the existing tips suggested here.

Generating reports: A good strategy for generating reports involves the combination of Sweave, make, and R.

Editor: Good editors for preparing Sweave documents include:

  • StatET and Eclipse
  • Emacs and ESS
  • Vim and Vim-R
  • R Studio

Code organisation: In terms of code organisation, I find two strategies useful:

user_1808920

I use project templates along with R studio, currently mine contains the following folders:

  • info : pdfs, powerpoints, docs... which won't be used by any script
  • data input : data that will be used by my scripts but not generated by them
  • data output : data generated by my scripts for further use but not as a proper report.
  • reports : Only files that will actually be shown to someone else
  • R : All R scripts
  • SAS : Because I sometimes have to :'(

I wrote custom functions so I can call smart_save(x,y) or smart_load(x) to save or load RDS files to and from the data output folder (files named with variable names) so I'm not bothered by paths during my analysis.

A custom function new_project creates a numbered project folder, copies all the files from the template, renames the RProj file and edits the setwd calls, and set working directory to new project.

All R scripts are in the R folder, structured as follow :

00_main.R
  • setwd
  • calls scripts 1 to 5
00_functions.R
  • All functions and only functions go there, if there's too many I'll separate it into several, all named like 00_functions_something.R, in particular if I plan to make a package out of some of them I'll put them apart
00_explore.R
  • a bunch of script chunks where i'm testing things or exploring my data
  • It's the only file where i'm allowed to be messy.
01_initialize.R
  • Prefilled with a call to a more general initialize_general.R script from my template folder which loads the packages and data I always use and don't mind having in my workspace
  • loads 00_functions.R (prefilled)
  • loads additional libraries
  • set global variables
02_load data.R
  • loads csv/txt xlsx RDS, there's a prefilled commented line for every type of file
  • displays which files hava been created in the workspace
03_pull data from DB.R
  • Uses dbplyr to fetch filtered and grouped tables from the DB
  • some prefilled commented lines to set up connections and fetch.
  • Keep client side operations to bare minimum
  • No server side operations outside of this script
  • Displays which files have been created in the workspace
  • Saves these variables so they can be reloaded faster

Once it's been done once I switch off a query_db boolean and the data will reloaded from RDS next time.

It can happen that I have to refeed data to DBs, If so I'll create additional steps.

04_Build.R
  • Data wrangling, all the fun dplyr / tidyr stuff goes there
  • displays which files have been created in the workspace
  • save these variables

Once it's been done once I switch off a build boolean and the data will reloaded from RDS next time.

05_Analyse.R
  • Summarize, model...
  • report excel and csv files
95_build ppt.R
  • template for powerpoint report using officer
96_prepare markdown.R
  • setwd
  • load data
  • set markdown parameters if needed
  • render
97_prepare shiny.R
  • setwd
  • load data
  • set shiny parameters if needed
  • runApp
98_Markdown report.Rmd
  • A report template
99_Shiny report.Rmd
  • An app template
user_22704750

I also do what Josh Reich does, only I do that creating my personal R-packages, as it helps me structure my code and data, and it is also quite easy to share those with others.

  1. create my package
  2. load
  3. clean
  4. functions
  5. do

creating my package: devtools::create('package_name')

load and clean: I create scripts in the data-raw/ subfolder of my package for loading, cleaning, and storing the resulting data objects in the package using devtools::use_data(object_name). Then I compile the package. From now on, calling library(package_name) makes these data available (and they are not loaded until necessary).

functions: I put the functions for my analyses into the R/ subfolder of my package, and export only those that need to be called from outside (and not the helper functions, which can remain invisible).

do: I create a script that uses the data and functions stored in my package. (If the analyses only need to be done once, I can put this script as well into the data-raw/ subfolder, run it, and store the results in the package to make it easily accessible.)

user_51240020
Related