How should I optimally choose bucket length in Spark's LSH algorithm Bucketed Random Projection?
It is this parameter set with .setBucketLength(), e.g. from the Spark documentation:
val brp = new BucketedRandomProjectionLSH()
.setBucketLength(2.0)
.setNumHashTables(3)
.setInputCol("keys")
.setOutputCol("values")
Any theoretical references that support the choice of bucket length would be appreciated.
The only suggestion I was able to find on the Web was this one, which does not explain the choice of the chosen heuristics.
To specify the value of bucketLength, if input vectors are normalized, 1-10 times of pow(numRecords, -1/inputDim) would be a reasonable value.
Therefore I am not really sure why that heuristics is supposed to work in general.