Background/Overview
I do a lot of work post-processing .csv files generated from different test setups/equipment. Often the post-processing tasks are the same/very similar from setup to setup and I find that I end up with scripts that are tailored to do just the specific operations for that setup. When a new script is needed it gets cobbled together from other existing scripts and I'll re-visit a setup and find that improvements I've made elsewhere aren't in the earlier scripts.
I would like to have a common post-processing "engine" from the code I have that can be used for all of my .csv post-processing needs, where the tool simply needs to load a configuration file specific to that setup to know what operations to perform (and on what fields). I am looking for recommendations/best practice for having this type of configuration file for this use case. Also note that the output data files from this will also be .csv. This is for compatibility with different tools that we have.
The config file should:
- Be easy to create using either a text editor or a special purpose tool (preferably free). Ideally someone can create a config file without having to know the underlying script details if they are given a manual about the options
- Allow the user to chain together operations sequentially (ex: apply a scaling factor, low pass filter, and then use as an input to calculate a new data column)
- Be scalable to easily support new features as they are developed without breaking backwards compatibility
Common operations include:
- Adding a column for elapsed time when only an absolute timestamp column exists
- Unit conversion
- Calculating power (P=V*I) or resistance (R=V/I) from measured voltage/current
- Resampling/Interpolating data from two different files that are sampled at different rates
- Synchronizing data from different files with different timestamp offsets
- Conversion of resistance measurements of thermistors into temperature (using Steinhart-Hart equation)
- Applying low-pass filters to data
Things I have considered
Talking with some colleagues they've recommended JSON and Python Configparser. I've done some very basic things with JSON for using default arguments with the Python Gooey library for remembering default values but not for anything more complicated. I haven't used configparser before, it seems like it could be a valid option but I want to get feedback before I go down a particular path