R / Python - Split String Column into multiple distinct Columns

Viewed 158

I have a string column in the format \

ID  Month 
1   Jan;Feb 
2   Mar;Apr;Jun;Jul;Aug;Nov 
3   Jan;May;Oct;Dec 
4   Apr;May;Sep;Oct 

I want to create unique/distinct 12 columns separated for each month like \

ID | M_1  | M_2 | M_3 | M_4 | M_5 | M_6 | M_7 | M_8 | M_9 | M_10 | M_11 | M_12 
1  | Jan  | Feb | NA  | NA  | NA  | NA  | NA  | NA  | NA  | NA   | NA   | NA 
2  |  NA   | NA  | Mar | Apr | NA  | Jun | Jul | Aug | NA  | NA   | Nov  | NA 
3  | Jan  | NA  | NA  | NA  | May | NA  | NA  | NA  | NA  | Oct  | NA   | Dec 
4  | NA   | NA  | NA  | Apr | May | NA  | NA  | NA  | Sep | Oct  | NA   | NA 

Would appreciate if someone can tell me to do the same in Python too.

(Apologies for the bad formatting)

6 Answers

Use Series.str.get_dummies and then multiply to get the names in the columns instead of 1/0 dummies. Rename and reorder the columns, then join the result back.

df1 = df['Month'].str.get_dummies(';')
df1 = df1.multiply(df1.columns).replace('', np.NaN)

# Or create the dict manually
mnths = pd.date_range('2010-01-01', '2010-12-31', freq='MS').strftime('%b')
d = {m: f'M_{i}' for m,i in zip(mnths, range(1, len(mnths)+1))}
#{'Jan': 'M_1', 'Feb': 'M_2', 'Mar': 'M_3', ... , 'Nov': 'M_11', 'Dec': 'M_12'}    

pd.concat([df[['ID']], df1.reindex(mnths, axis=1).rename(columns=d)], axis=1)

   ID  M_1  M_2  M_3  M_4  M_5  M_6  M_7  M_8  M_9 M_10 M_11 M_12
0   1  Jan  Feb  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN  NaN
1   2  NaN  NaN  Mar  Apr  NaN  Jun  Jul  Aug  NaN  NaN  Nov  NaN
2   3  Jan  NaN  NaN  NaN  May  NaN  NaN  NaN  NaN  Oct  NaN  Dec
3   4  NaN  NaN  NaN  Apr  May  NaN  NaN  NaN  Sep  Oct  NaN  NaN

This is a Python solution :

Step 1 : Create a pairing of the months :

months = ('Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun',
          'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec')

months_column = [f"M_{index}" for index, _ in enumerate(months, 1)]

print(months_columns)
['M_1', 'M_2', 'M_3', 'M_4', 'M_5', 'M_6', 'M_7', 'M_8', 'M_9', 'M_10', 'M_11', 'M_12']

#pair the months with the months_column : 
mapping = dict(zip(months, months_column))
print(mapping)
{'Jan': 'M_1', 'Feb': 'M_2', 'Mar': 'M_3', 'Apr': 'M_4', 'May': 'M_5', 'Jun': 'M_6',
 'Jul': 'M_7', 'Aug': 'M_8', 'Sep': 'M_9', 'Oct': 'M_10', 'Nov': 'M_11', 'Dec': 'M_12'}

Step 2 : move into Pandas realm :

  #import pandas as pd

   df = pd.read_csv(filename) #
  
   df["Month"] = df.Month.str.split(";")
   #expand each word in the Month column to separate rows
   df = df.explode("Month")
   df["Month_column"] = df.Month.map(mapping)
   #final step is to pivot the dataframe to get your result
   df.pivot(index="ID", columns="Month_column", values="Month").reindex(columns=months_column)
  • You have a specified output, but I'm presenting this as an alternative.
  • Use .str.split to separate the strings at ;
  • Use .explode do convert the list values to separate rows
  • Use pandas.get_dummies to convert categorical variable into dummy/indicator variables.
  • Use .groupby and aggregate .sum to combine the indicators for each month by ID
import pandas as pd
import calendar  # to get month abbreviations in order

month_abr = list(calendar.month_abbr)

# test data
data = {'ID': [1, 2, 3, 4],
        'Month': ['Jan;Feb', 'Mar;Apr;Jun;Jul;Aug;Nov', 'Jan;May;Oct;Dec', 'Apr;May;Sep;Oct']}

# setup dataframe
df = pd.DataFrame(data)

# convert month rows to lists
df.Month = df.Month.str.split(';')

# explode lists into rows
df = df.explode('Month')

# create indicators and groupby sum
dfi = pd.get_dummies(df, prefix='', prefix_sep='').groupby('ID', as_index=False).sum()

# optionally reorder the columns
dfi = dfi[dfi.columns.reindex(['ID'] + month_abr[1:])[0]]

# display(dfi)
   ID  Jan  Feb  Mar  Apr  May  Jun  Jul  Aug  Sep  Oct  Nov  Dec
0   1    1    1    0    0    0    0    0    0    0    0    0    0
1   2    0    0    1    1    0    1    1    1    0    0    1    0
2   3    1    0    0    0    1    0    0    0    0    1    0    1
3   4    0    0    0    1    1    0    0    0    1    1    0    0

More concisely

# test data
data = {'ID': [1, 2, 3, 4],
        'Month': ['Jan;Feb', 'Mar;Apr;Jun;Jul;Aug;Nov', 'Jan;May;Oct;Dec', 'Apr;May;Sep;Oct']}

# setup dataframe
df = pd.DataFrame(data)

# get indicators and concat with the IDs
dfi = pd.concat([df.ID, df.Month.str.get_dummies(';')], axis=1)

# optionally reorder the columns
dfi = dfi[dfi.columns.reindex(['ID'] + month_abr[1:])[0]]

# display(dfi)
   ID  Jan  Feb  Mar  Apr  May  Jun  Jul  Aug  Sep  Oct  Nov  Dec
0   1    1    1    0    0    0    0    0    0    0    0    0    0
1   2    0    0    1    1    0    1    1    1    0    0    1    0
2   3    1    0    0    0    1    0    0    0    0    1    0    1
3   4    0    0    0    1    1    0    0    0    1    1    0    0

We can get data in long format, create a column with index, arrange the data and get it back in wide format. Using dplyr and tidyr this can be done as :

library(dplyr)  
library(tidyr)

df %>%
    separate_rows(Month, sep = ";") %>%
    mutate(col = paste0('M_', match(Month, month.abb))) %>%
    arrange(match(Month, month.abb)) %>%
    pivot_wider(names_from = col, values_from = Month) %>%
    arrange(ID)

# A tibble: 4 x 13
#     ID M_1   M_2   M_3   M_4   M_5   M_6   M_7   M_8   M_9   M_10  M_11  M_12 
#  <int> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
#1     1 Jan   Feb   NA    NA    NA    NA    NA    NA    NA    NA    NA    NA   
#2     2 NA    NA    Mar   Apr   NA    Jun   Jul   Aug   NA    NA    Nov   NA   
#3     3 Jan   NA    NA    NA    May   NA    NA    NA    NA    Oct   NA    Dec  
#4     4 NA    NA    NA    Apr   May   NA    NA    NA    Sep   Oct   NA    NA   

data

df <- structure(list(ID = 1:4, Month = c("Jan;Feb", "Mar;Apr;Jun;Jul;Aug;Nov", 
"Jan;May;Oct;Dec", "Apr;May;Sep;Oct")), class = "data.frame", 
row.names = c(NA, -4L))

Here's a way to do this in R. I split your string on the semicolon and used a named vector of the months with each column containing NA values. I iterated through the split string list to get each value and replace the NA values with the string value based on the name.

df <- data.frame(
            ID = c(1L, 2L, 3L, 4L),
         Month = c("Jan;Feb","Mar;Apr;Jun;Jul;Aug;Nov",
                   "Jan;May;Oct;Dec","Apr;May;Sep;Oct")
  )
  
  
  parse_dataframe <- function(string, splt=';'){
    splt_string <- strsplit(string, splt)
    output <- list()
    count = 1
    for (i in splt_string){
      key <- rep(NA, 12)
      names(key) <- c('Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec')
      for (month in i){
        key[month] <- month
      }
      output[[count]] <- key
      count <- count + 1
    }
    output <- do.call(rbind, output)
    colnames(output) <- paste0('M_', seq(1,12))
    return(output)
  }
  
  parse_dataframe(df$Month)

Here's something similar with python where your data is df and the column is 'Month'. I used pandas because there's not really a native way to get NA values with the standard libraries.

import pandas as pd
import calendar 

def parse_dataframe(string_series, splt=';'):
    str_splt = string_series.str.split(splt)
    months = [calendar.month_abbr[x] for x in range(1, 13)]
    output = []

    for row in str_splt:
        output_row = [pd.NA for _ in range(12)]
        for month in row:
            output_row[months.index(month)] = month

        output.append(output_row)
    df = pd.DataFrame(output)
    df.columns = ['M_' + str(x) for x in range(1, 13)]

    return df

parse_dataframe(df['Month'])

Here is a more compact base R solution:

df <- data.frame(
  ID = c(1:4),
  Month = c("Jan;Feb","Mar;Apr;Jun;Jul;Aug;Nov",
            "Jan;May;Oct;Dec","Apr;May;Sep;Oct")
)

Months <- factor(month.abb, month.abb)
dfm <- do.call(rbind, lapply(strsplit(df$Month, ";"), 
  function(x) x[Months][match(Months, x)]))
setNames(data.frame(df$ID, dfm), c("ID", paste0("M_", seq_along(Months))))
#>   ID  M_1  M_2  M_3  M_4  M_5  M_6  M_7  M_8  M_9 M_10 M_11 M_12
#> 1  1  Jan  Feb <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA> <NA>
#> 2  2 <NA> <NA>  Mar  Apr <NA>  Jun  Jul  Aug <NA> <NA>  Nov <NA>
#> 3  3  Jan <NA> <NA> <NA>  May <NA> <NA> <NA> <NA>  Oct <NA>  Dec
#> 4  4 <NA> <NA> <NA>  Apr  May <NA> <NA> <NA>  Sep  Oct <NA> <NA>

Created on 2020-08-08 by the reprex package (v0.3.0)

Related