python: convert a list of (string) sets to a scipy csr_matrix

user_3659634 0
Viewed 135

Suppose I have this list of sets:

db = [{"bread", "butter", "milk"}, 
      {"eggs", "milk", "yogurt"},
      {"bread", "cheese", "eggs", "milk"}, 
      {"eggs", "milk", "yogurt"},
      {"cheese", "milk", "yogurt"}]

How do I convert this into a scipy sparse csr_matrix? Its' expected output is the following:

[[1., 1. 0., 0., 1., 0.],
 [0., 0., 0., 1., 1., 1.],
 [1., 0., 1., 1., 1., 0.],
 [0., 0., 0., 1., 1., 1.],
 [0., 0., 1., 0., 1., 1.]]

I tried hardcoding it so I could digest it further but i can't seem to understand. My code is:

indptr = np.array([0, 3, 6, 10, 13, 16])
data = np.array(["bread", "butter", "milk", "eggs", "milk", "yogurt",
                "bread", "cheese", "eggs", "milk","eggs", "milk", "yogurt",
                "cheese", "milk", "yogurt"])
indices = np.array([0, 1, 4, 3, 4, 5, 0, 2, 3, 4, 3, 4, 5, 2, 4, 5])
csr_matrix((data, indices, indptr), dtype=int).toarray()

I can't seem to make it work. Is there a better way of implementing this?

scipy sparse-matrix python
1 Answers

Setup:

import pandas as pd
import numpy as np
from scipy.sparse import csr_matrix

db = [{"bread", "butter", "milk"}, 
      {"eggs", "milk", "yogurt"},
      {"bread", "cheese", "eggs", "milk"}, 
      {"eggs", "milk", "yogurt"},
      {"cheese", "milk", "yogurt"}]

all_products = set()
for SET in db:
    all_products |= SET
sorted_products = sorted(all_products)

Method 2 (no pandas):

First, you make translator

d = dict()
for i, prod in enumerate(sorted_products):
    d[prod] = i

{'bread': 0, 'butter': 1, 'cheese': 2, 'eggs': 3, 'milk': 4, 'yogurt': 5}

Then, you make full matrix and populate it

template = np.zeros(len(all_products) * len(db), dtype=int).reshape((len(db), len(all_products)))
for j, line in enumerate(db):
    for prod in line:
        template[j, d[prod]] = 1

array([[1, 1, 0, 0, 1, 0],
       [0, 0, 0, 1, 1, 1],
       [1, 0, 1, 1, 1, 0],
       [0, 0, 0, 1, 1, 1],
       [0, 0, 1, 0, 1, 1]])

and lastly convert it to sparse matrix

matrix = csr_matrix(template)

  (0, 0)    1
  (0, 1)    1
  (0, 4)    1
  (1, 3)    1
  (1, 4)    1
  (1, 5)    1
  (2, 0)    1
  (2, 2)    1
  (2, 3)    1
  (2, 4)    1
  (3, 3)    1
  (3, 4)    1
  (3, 5)    1
  (4, 2)    1
  (4, 4)    1
  (4, 5)    1

#<5x6 sparse matrix of type '<class 'numpy.longlong'>'
#   with 16 stored elements in Compressed Sparse Row format>

Method 1 (pandas):

df = pd.DataFrame(index=sorted_products, columns=range(len(db)))
print(df)

Gives you empty dataframe

          0       1       2       3       4
yogurt  NaN     NaN     NaN     NaN     NaN
butter  NaN     NaN     NaN     NaN     NaN
bread   NaN     NaN     NaN     NaN     NaN
milk    NaN     NaN     NaN     NaN     NaN
cheese  NaN     NaN     NaN     NaN     NaN
eggs    NaN     NaN     NaN     NaN     NaN

Then you add sets

for i in range(len(db)):
    df[i] = pd.Series([1]*len(db[i]), index=list(db[i]))

          0       1       2       3       4
yogurt  NaN     1.0     NaN     1.0     1.0
butter  1.0     NaN     NaN     NaN     NaN
bread   1.0     NaN     1.0     NaN     NaN
milk    1.0     1.0     1.0     1.0     1.0
cheese  NaN     NaN     1.0     NaN     1.0
eggs    NaN     1.0     1.0     1.0     NaN

Next, you fill NaN values with zeroes

data = df.fillna(0)

And at the end you convert it to sparse matrix

from scipy.sparse import csr_matrix
matrix = csr_matrix(data)
print(matrix)

Outputs:

#<6x5 sparse matrix of type '<class 'numpy.longlong'>'
#   with 16 stored elements in Compressed Sparse Row format>
  (0, 2)    1
  (0, 4)    1
  (1, 1)    1
  (1, 2)    1
  (1, 3)    1
  (2, 0)    1
  (2, 1)    1
  (2, 2)    1
  (2, 3)    1
  (2, 4)    1
  (3, 1)    1
  (3, 3)    1
  (3, 4)    1
  (4, 0)    1
  (4, 2)    1
  (5, 0)    1
user_86644220
Related