Partial k-means Clustering - considering missing data

Question

I have a big data frame, I have to use specific 8 columns from the data frame where the values are either "strongly agree", "agree" or "disagree". Depending on these 8 columns, I need to create a new column which tell which cluster(1-8) the row belongs to (preferably by using K-means clustering). But My data also has NaN values. I want to do the clustering considering the NaN values i.e., partial k-means clustering. (please refer the dataset below)

This is because:

There is an already existing code in SPSS where clustering in done by setting \MISSING = PAIRWISE. I am trying to convert this code to pyspark because of technical reasons. I could not find any alternative for pairWISE in pyspark( Which is nothing but doing partial k-means cluster analysis) Previously clusters were made by using this PAIRWISEcode and now I have no other alternative but convert it to pyspark.

PS: I am new to PySpark, Any help would be highly appreciated

+--------+--------+--------+--------+--------+--------+--------+--------+----------+
|    col1|    col2|    col3|    col4|    col5|    col6|    col7|    col8| Cluster  |
+--------+--------+--------+--------+--------+--------+--------+--------+----------+
|NaN     |disagree|disagree|NaN     |disagree|disagree|disagree|disagree|         2|
|disagree|disagree|disagree|disagree|disagree|disagree|disagree|disagree|         1|
|disagree|NaN     |disagree|disagree|NaN     |NaN     |disagree|disagree|         5|
[...]
|disagree|   agree|disagree|   agree|   agree|disagree|disagree|disagree|         3|
|disagree|disagree|disagree|disagree|disagree|disagree|disagree|disagree|         1|
|disagree|disagree|disagree|disagree|disagree|disagree|   agree|disagree|         5|
|disagree|   agree|   agree|   agree|disagree|disagree|disagree|   agree|         3|
|   agree|   agree|   agree|disagree|disagree|   agree|disagree|disagree|         6|
[...]
|   agree|   agree|   agree|   agree|   agree|   agree|   agree|   agree|         7|
|   agree|   agree|   agree|   agree|   agree|disagree|   agree|   agree|         2|
|   agree|   agree|   agree|   agree|   agree|   agree|   agree|   agree|         7|
+--------+--------+--------+--------+--------+--------+--------+--------+----------+

The spss code is as follows:

QUICK CLUSTER 
col1
col2
col3
col4
col5
col6
col7
col8
  /MISSING=pairWISE
  /CRITERIA=CLUSTER(8) 
  /METHOD=CLASSIFY
  /SAVE CLUSTER
  /PRINT INITIAL
  /FILE='file.sav'.

Value labels Q_Cluster 1 "Cluster 1" 2 "Cluster 2" 3 "Cluster 3" 4 "Cluster 4"
    5 "Cluster 5" 6 "Cluster 6" 7 "Cluster 7" 8 "Cluster 8".
Execute.

*Exclusion of cases with too high a number (>4) of missing values in the cluster-forming variables; name of the cluster variables behind 'Recode' may have to be adjusted.
COUNT Miscount=col1 col2 col3
    col4 col5 col6 col7
    col8(SYSMIS).
VARIABLE LABELS  Miscount 'Number of missings in cluster image. Var.'.
DO IF (Miscount>4).
RECODE Q_Cluster (1 thru 8=SYSMIS).
END IF.
EXECUTE.

Any ideas on how to exclude the NaN cases would be really helpful

user_10445333 · Answer

Based on my understanding, K-means clustering in Pyspark doesn't support so-called "partial clustering", i.e., every row in every column should be a quantitive value. Therefore I believe you have to modify the spark code to achieve the same goal in SPSS.

Actually, this will trigger another question. Based on my understanding, K-means clustering uses Euclidean distance to calculate, which means all values in each row should be a quantitive value. Based on the example dataframe that you provided, how do you want to OR how does SPSS handle the null value imputation? Do you have any references or papers so that we can transfer the methodology by using pyspark?

Partial k-means Clustering - considering missing data

1 Answers