How to cluster large amounts of data with minimal memory usage

You'll need to choose a different algorithm.

Hierarchical clustering needs O(n²) memory and the textbook algorithm O(n³) time. This cannot scale well to large data.

You could have a look at

• DBSCAN (or other density-based algorithms) and perhaps this associated discussion: scikit-learn DBSCAN memory usage
• SLINK, a variation of hierarchical clustering (set linkage='single' in sklearn.cluster.AgglomerativeClustering)
• MiniBatch K Means
• or BIRCH.

However, you will have to set up some pipeline to test different numbers of clusters. It's hard to say which algorithm will work best for you, though.