I have a list of ~381,000 strings. I managed to calculate the string distance for all NC2 pairs (~72 billion).
Now, I would like to form clusters of strings for quick search. I usually use sklearn agglomerative clustering. But, I fear this time I might face memory issues.
The purpose of the string cluster is so that I can quickly search similar names when the Data Entry Executive enters a name.
How can I achieve clustering on such a large set? Are there better ways to achieve my purpose?