Issues converting lists from massive dictionary to dataframe

user_7991624 0
Viewed 230

I have created a dictionary in this way:

The data looks like this:

GDS3:
ABC_1     ABC_2     BBB_1
cat        elf       123
dog        run       456
bird       burp      789

GDS4:
ABC_3     ABC_4     BCB_a
beer        yes      234
wine        no       543
gin         yes      743

GDS5:
ABC_5     ABC_6     BCD_c
lol        yea       543
lmao       NaN       446
asl        NaN       777

#create a dictionary in which all columns that start with the same 3 characters will be grouped in the same key. 
dict_2013 = {k: g for k, g in GDS3.groupby(by=lambda x: x[:3].lower(), axis=1)}

dict_2014 = {k: g for k, g in GDS4.groupby(by=lambda x: x[:3].lower(), axis=1)}

dict_2015 = {k: g for k, g in GDS5.groupby(by=lambda x: x[:3].lower(), axis=1)}

#start with year 2013:
global_dict=dict_2013

#if key in the new dictionary is in the old dictionary then 
#add the values from the new dictionary key to the old dictionary key
#else if the new dictionary key does not exist in the old dictionary then add a new key with the new values

for key,val in dict_2014.items():
    if key in global_dict:
       global_dict[key]=[global_dict[key],val]
    else:
       global_dict[key]=val

for key,val in dict_2015.items():#to add items
    if key in global_dict:
        global_dict[key]=[global_dict[key],val]
    else:
       global_dict[key]=val

This is my desired output (a dataframe for each key)

  df_ABC:
  ABC_1     ABC_2     ABC_3   ABC_4   ABC_5
  cat        elf       beer    yes    lol
  dog        run       win     no     lmao
  bird       burp      gin     yes    asl

  df_BBB:
  BBB_1
  cat   
  dog        
  bird

In other words,I want to convert the individual keys into individual dictionaries (FOR ALL OF THE KEYS), so I tried the following:

ABC_dataframe=pd.DataFrame(global_dict['ABC'])

When I do this I get the following error:

TypeError: Expected list, got DataFrame

Which is strange because global_dict['ABC'] is a list. (I checked using type(global_dict['ABC']).

What can I do to correct this? I tried flattening the list but I'm still having issues.

dataframe python
2 Answers

The most confusing part of your logic is having global_dict values either a dataframe or a list. Keep object types consistent; choose list and append to it each time you wish to add a value.

The Pythonic solution is to use a collections.defaultdict of list objects:

from collections import defaultdict

global_dict = defaultdict(list, {k: [v] for k, v in dict_2013.items()})

for key,val in dict_2014.items():
    global_dict[key].append(val)

for key,val in dict_2015.items():
    global_dict[key].append(val)

Then use pd.concat along axis=1:

abc = pd.concat(global_dict['abc'], axis=1)

print(abc)

  ABC_1 ABC_2 ABC_3 ABC_4 ABC_5 ABC_6
0   cat   elf  beer   yes   lol   yea
1   dog   run  wine    no  lmao   NaN
2  bird  burp   gin   yes   asl   NaN

I cannot explain why your desired result is missing ABC_6.

user_92095460

You could do it using pd.concat and groupby, if GDS3, GDS4, and GSD5 are already dataframes:

tdf = pd.concat([GDS3, GDS4, GDS5], axis=1)

g = tdf.groupby(tdf.columns.str[:3], axis=1)

# Now, let's create a dictionary of dataframes grouped 
# by the first three letters of each column.

df_list = {}
for n, i in g:
    df_list[n] = i


print(df_list['ABC'])
print(df_list['BBB'])

Or as @jpp suggests use:

dict_dfs = dict(tuple(g))

print(dict_dfs['ABC'])
print(dict_dfs['BBB'])

Output:

  ABC_1 ABC_2 ABC_3 ABC_4 ABC_5 ABC_6
0   cat   elf  beer   yes   lol   yea
1   dog   run  wine    no  lmao   NaN
2  bird  burp   gin   yes   asl   NaN
   BBB_1
0    123
1    456
2    789
user_63615310
Related