Generate a sparse dataframe from a list of index positions and values

Viewed 68

New learner here. I have a list of data values that are labeled by a comma-delimited string that represents the position in a dataframe; think of the string as representing the row (say 1-20) and column (say A-L) index values of a position in the array where the corresponding value should go. The populated data frame would be sparse, with many empty cells. I am working with pandas for the first time on this project, and am still learning the ropes.

position value

1,A 32

1,F 16

2,B 234

2,C 1345

2,E 13

2,G 999

3,D 5332

4,B 12

etc.

I have been trying various approaches, but am not satisfied. I created dummy entries for empty cells in the completed dataframe, then iterated over the list to write the value to the correct cell. It works but it is not elegant and it seems like a brittle solution.

I can pre-generate a dataframe and populate it, or generate a new dataframe as part of the population process: either solution would be fine. It seems like this should be a simple task. Maybe even a one liner! But I am stumped. I would appreciate any pointers.

2 Answers

As you suggest, the simplest method might be a for-loop to initialize the non-empty values. Alternatively, you can use pivot() or numpy advanced indexing. All options are shown below.

The only tricky thing is ensuring that your dataframe result will have the complete set of rows and columns, as explained in the update below.

text = """\
row,column,value
1,A 32
1,F 16
2,B 234
2,C 1345
2,E 13
2,G 999
3,D 5332
4,B 12
"""
from io import StringIO
import numpy as np
import pandas as pd

# Load your data and convert the column letters to integers.
# Note: Your exapmple data is delimited with both spaces and commas,
#       which is why we need a custom 'sep' argument here.
entries = pd.read_csv(StringIO(text), sep='[ ,]+')
entries['icol'] = entries['column'].map(lambda c: ord(c) - ord('A'))

# Construct an empty DataFrame with the appropriate index and columns.
rows = range(1, 1 + entries['row'].max())
columns = [chr(ord('A') + i) for i in range(1 + entries['icol'].max())]
df = pd.DataFrame(index=rows, columns=columns)

##
## Three ways to populate the dataframe:
##

# Option 1: Iterate in a for-loop
for e in entries.itertuples():
    df.loc[e.row, e.column] = e.value

# Option 2: Use pivot() or unstack()
df = df.fillna(entries.pivot('row', 'column', 'value'))

# Option 3: Use numpy indexing to overwrite the underlying array:
irows = entries['row'].values - 1
icols = entries['icol'].values
df.values[irows, icols] = entries['value'].values

Result:

     A    B     C     D    E    F    G
1   32  NaN   NaN   NaN  NaN   16  NaN
2  NaN  234  1345   NaN   13  NaN  999
3  NaN  NaN   NaN  5332  NaN  NaN  NaN
4  NaN   12   NaN   NaN  NaN  NaN  NaN

Update:

Late in the day, it occurred to me that this can be solved via pivot() (or unstack(), as suggested by @piterbarg). I've now included that option above.

In fact, it's tempting to just use pivot() without pre-initializing the DataFrame. HOWEVER, there's an important caveat to that approach: If any particular row or column value remains completely unused in your original entries data, then those rows will remain completely omitted from the final table. That is, if no entry uses row 3, your final table would only contain rows 1,2,4. Likewise, if your data contains no data for columns C,E,G (for example), then you would end up with columns A,B,D,F.

If you want to be sure that your rows use contiguous index values and your columns use a contiguous sequence of letters, then pivot() or unstack() is not enough. You must first initialize the indexes of your dataframe as shown above.

This is a standard unstack:

entries.set_index(['row','column']).unstack()

where entries is defined in @StuartBerg answer:

entries = pd.read_csv(StringIO(text), sep='[ ,]+')

output:

    value
column  A    B      C        D      E     F     G
row                         
1       32.0 NaN    NaN     NaN     NaN   16.0  NaN
2       NaN  234.0  1345.0  NaN     13.0  NaN   999.0
3       NaN  NaN    NaN     5332.0  NaN   NaN   NaN
4       NaN  12.0   NaN     NaN     NaN   NaN   NaN
Related