How to manipulate and slice multi-dimensional JSON data in Python?

Viewed 24

I'm trying to set up a convenient system for storing and analyzing data from experiments. For the data files I use the following JSON format:

{
    "sample_id": "",
    "timestamp": "",
    "other_metadata1": "",

    "measurements": {
        "type1": {
            "timestamp": "",
            "other_metadata2": ""
            
            "data": {
                "parameter1": [1,2,3],
                "parameter2": [4,5,6]
            }
        }

        "type2": { ... }
    }
}

Now for analyzing many of these files, I want to filter for sample metadata and measurement metadata to get a subset of the data to plot. I wrote a function like this:

def get_subset(data_dict, include_samples={}, include_measurements={}):
    # Start with a copy of all datasets
    subset = copy.deepcopy(data_dict)

    # Include samples if they satisfy certain properties
    for prop, req in include_samples.items():
        subset = {file: sample for file, sample in subset.items() if sample[prop] == req}
    
    # Filter by measurement properties
    for file, sample in subset.items():
        measurements = sample['measurements'].copy()
        
        for prop, req in include_measurements.items():
            measurements = [meas for meas in measurements if meas[prop] == req]
        
        # Replace the measurements list
        sample['measurements'] = measurements
    
    return subset

While this works, I feel like I'm re-inventing the wheel of something like pandas. I would like to have more functionality like dropping all NaN values, excluding based on metadata, etc, All of which is available in pandas. However my data format is not compatible with the 2D nature of that.

Any suggestions on how to go about manipulating and slicing such data strutures without reinventing a lot of things?

0 Answers
Related