I am using scipy library for kmeans algorithm. My dataset was 81k and K was 8613. I used my own algorithm to calculate K. I am not saying my algorithm for choosing K is perfect. My concern is not to chose optimal K.
My main concern is the final result. After running scipy kmeans algorithm I get the final result with 6995 number cluster. Here, around 1600 K is not used. Let's say the number of K is given more and based on that is scipy K-means algorithm not choosing excess K.
Can anyone please explain why not using all of K?