Fuzzy C-Means Clustering in R

This is an attempt at an answer, based on limited information and it may not fully address the questioners situation. It sounds like there may be other issues. In chat they indicated that they had encountered additional errors that I can not reproduce. Fanny will calculate and assign items to "crisp" clusters, based on a metric. It will also produce a matrix showing the fuzzy clustering assignment that may be accessed using membership.

The issue the questioner described can be recreated by increasing the memb.exp parameter using the iris data set. Here is an example:

library(plyr)
library(clValid)
library(cluster)
df<-iris[,-5] # I do not use iris, but to make reproducible
clust<-sapply(df,scale)

res.fanny <- fanny(clust, 4, metric='SqEuclidean', memb.exp = 2) 

Calling res.fanny$k.crisp shows that this produces 4 crisp clusters.

res.fanny14 <- fanny(clust, 4, metric='SqEuclidean', memb.exp = 14)

Calling res.fanny14$k.crisp shows that this produces 3 crisp clusters.

One can still access the membership of each of the 4 clusters using res.fanny14$membership.

If you have a good reason to think there should be 4 crisp clusters one could reduce the memb.exp parameter. Which would tighten up the cluster assignments. Or if you are doing some sort of supervised learning, one procedure to tune this parameter would be to reserve some test data, do a hyperparameter grid search, then select the value that produces the best result on your preferred metric. However without knowing more about the task, the data, or what the questioner is trying to accomplish it is hard to suggest much more than this.

First of all I encourage to read the nice vignette of the clValid package. The R package clValid contains functions for validating the results of a cluster analysis. There are three main types of cluster validation measures available. One of this measure is the Dunn index, the ratio between observations not in the same cluster to the larger intra-cluster distance. I focus on Dunn index for simplicity. In general connectivity should be minimized, while both the Dunn index and the silhouette width should be maximized. clValid creators explicitly refer to the fanny function of the cluster package in their documentation.

The clValid package is useful for running several algorithms/metrics across a prespecified sets of clustering.

library(dplyr)
library(clValid)
iris
table(iris$Species)
clust <- sapply(iris[, -5], scale)

In my code I need to increase the iteration for reaching convergence (maxit = 1500). Results are obtained with summary function applied to the clValid object intvalid. Seems that the optimal number of clusters is 2 (but here is not the main point).

 intvalid <- clValid(clust, 2:5, clMethods=c("fanny"), 
                    maxit = 1500,
                    validation="internal",
                    metric="euclidean")
 summary(intvalid)

The results from any method can be extracted from a clValid object for further analysis using the clusters method. Here the results from the 2 clusters solution are extracted(hc$2), with emphasis on the Dunnett coefficient (hc$2$coeff). Of course this results were related to the "euclidean" metric of the clValid call.

hc <- clusters(intvalid, "fanny")
hc$`2`$coeff

Now, I simply call fanny from cluster package using euclidean metric and 2 clusters. Results are completely overlapping with the previous step.

res.fanny <- fanny(clust, 2, metric='euclidean', maxit = 1500)
res.fanny$coeff

Now, we can look at the classification table

 table(hc$`2`$clustering, iris[,5])

    setosa versicolor virginica
  1     50          0         0
  2      0         50        50

and to the profile

df$fuzzy <- hc$`2`$clustering
profile <- ddply(df,.(fuzzy), summarize,
               count=length(fuzzy))
profile
  fuzzy count
1     1    50
2     2   100