How to reference a global config file for filtering data in Python 3.7 / Pandas?

user_16735606 0
Viewed 137

I know this is possible due to having a Python script that performed this filtering function at my old job but I sadly left some files behind on my transition to a new role. I am not able to get these back so I have been struggling to figure out how to reference a config file that can be used to filter the final dataframe.

An example of the global config.py file in question looks like this:

global_config_eam = {

'athena_mktg_segment': 'MID-MARKET',
'cmcst_status': 'NEVER',
'mdt_sellable_green_yellow_coax_ind': 1,
'nat_account_ind': 0,
'athena_business_source_count': ['>',1],
'Phone_c_dnc': 'N',
'phone_in_csg': 'N',

}

The above will look at the dataframe in question and only keep rows that match the values in the columns above.

It is is referenced by another utility at the beginning which is where I am scratching my head. I know I would need to build another .py file with a def command to reference the config to filter but I cannot seem to find any articles on this. The example of the import section and use of both files is as follows:

from src.utils.utilities import filter_me, check_dnc, transform_df, DoNotCall_SFTP, rank_me_adhoc

...

df_scrubbed, df_waterfall, mc = filter_me(data_dnc_ind, global_config_eam, 'athena_business_id')

If anyone has any idea on how where to start with incorporating a config.py file using def filter_me (as an example) I would extremely grateful. I know this isn't much to go off of but I'm hoping someone out there has created something like this or has some articles that clearly fit the requirements of the function I am trying to recreate.

pandas dataframe filter config python
1 Answers

You could turn this dict into a query string that you can pass to your dataframe. You just have to make sure you get the logic right.

def make_query_string(k,v):
    if isinstance(v, list):
        return " ".join([k] + [str(x) for x in v])
    else:
        if isinstance(v, str):
            v = f"'{v}'"
        else:
            v = str(v)
        return " == ".join([k,v])

query_string = " and ".join([make_query_string(k,v) for k,v in global_config_eam.items()])

query_string will look like this

"athena_mktg_segment == 'MID-MARKET' and cmcst_status == 'NEVER' and mdt_sellable_green_yellow_coax_ind == 1 and nat_account_ind == 0 and athena_business_source_count > 1 and Phone_c_dnc == 'N' and phone_in_csg == 'N'"

You can then use it to filter

df.query(query_string)
user_114727610
Related