I'm trying to find similarity between text messages (about 1 million text message), in my implementation each line represents an entry.
In order to calculate similarity between those texts we adopt tfidf and columnSimilarities
Below is the code:
import org.apache.spark.rdd.RDD
import org.apache.spark.SparkContext
import org.apache.spark.mllib.feature.HashingTF
import org.apache.spark.mllib.linalg.Vector
import org.apache.spark.mllib.feature.IDF
import org.apache.spark.mllib.linalg.distributed.RowMatrix
import org.apache.spark.mllib.linalg.distributed.MatrixEntry
import org.apache.spark.mllib.linalg.distributed.IndexedRowMatrix
import org.apache.spark.mllib.linalg.distributed.IndexedRow
//import scala.math.Ordering
//import org.apache.spark.RangePartitioner
def transposeRowMatrix(m: RowMatrix): RowMatrix = {
val indexedRM = new IndexedRowMatrix(m.rows.zipWithIndex.map({
case (row, idx) => IndexedRow(idx, row)}))
val transposed = indexedRM.toCoordinateMatrix().transpose.toIndexedRowMatrix()
new RowMatrix(transposed.rows
.map(idxRow => (idxRow.index, idxRow.vector))
.sortByKey().map(_._2))
}
// split word based on spaces and special characters
val documents = sc.textFile("./test1").map(_.split(" |\\,|\\?|\\-|\\+|\\*|\\(|\\)|\\[|\\]|\\{|\\}|\\<|\\>|\\/|\\;|\\.|\\:|\\=|\\^|\\|").filter(_.nonEmpty).toSeq)
val hashingTF = new HashingTF()
val tf = hashingTF.transform(documents)
tf.cache()
println(tf.getNumPartitions)
val idf = new IDF().fit(tf)
val tfidf = idf.transform(tf)
val mat = new RowMatrix(tfidf)
// transpose matrix to get result between row (not between column)
val sim = transposeRowMatrix(mat).columnSimilarities()
val trdd = sim.entries.map{case MatrixEntry(row: Long, col:Long, sim:Double) => Array(row,col,sim).mkString(",")}
println(trdd.getNumPartitions)
// to descease write to file time
val transformedRDD = trdd.repartition(50)
println(transformedRDD.getNumPartitions)
transformedRDD.cache()
transformedRDD.saveAsTextFile("output")*
The problem is when the number of similar messages increases in file, the similarity decreases.
e.g.
let's assume that we have the file below:
hello world
hello world 123
how is every thing
we are testing
this is a test
corporate code 123-234 you ca also tap on this link to verify corporate.co/1234
corporate code 134-456 you ca also tap on this link to verify corporate.co/5667
The output of the former command is:
%cat output/part-000*
5.0,6.0,0.7373482646933146
0.0,1.0,0.8164965809277261
4.0,5.0,0.053913565847778636
1.0,5.0,0.13144171271256438
2.0,4.0,0.16888723050548915
4.0,6.0,0.052731941041749664
Each line in the output represents the similarity between two lines as follows: "lineX -1", "lineY -1", "similarity"
The output showing the similarity between the last 2 lines is 5.0,6.0,0.7373482646933146, which is fine.
The two lines are
corporate code 123-234 you ca also tap on this link to verify corporate.co/1234
corporate code 134-456 you ca also tap on this link to verify corporate.co/5667
and similarity is 0.7373482646933146
while when the file input is:
hello world
hello world 123
hello world 956248
hello world 2564
how is every thing
we are testing
this is a test
corporate code 123-234 you ca also tap on this link to verify corporate.co/1234
corporate code 134-456 you ca also tap on this link to verify corporate.co/5667
corporate code 456-458 you ca also tap on this link to verify corporate.co/8965
corporate code 444-444 you ca also tap on this link to verify corporate.co/4444
the output is:
7.0,10.0,0.4855543123154418
2.0,3.0,0.32317021425463427
6.0,8.0,0.03657892871242232
6.0,10.0,0.03097823353416634
0.0,1.0,0.6661166307685673
7.0,8.0,0.5733398760974173
1.0,2.0,0.37867439463004254
9.0,10.0,0.4855543123154418
0.0,3.0,0.5684806190668547
8.0,9.0,0.6716256614182469
4.0,6.0,0.1903502047647684
8.0,10.0,0.4855543123154418
1.0,3.0,0.37867439463004254
6.0,9.0,0.03657892871242232
7.0,9.0,0.5733398760974173
6.0,7.0,0.03657892871242232
1.0,7.0,0.233827426275723
0.0,2.0,0.5684806190668547
the output between the same lines tested in the first example is: 7.0,8.0,0.5733398760974173
the similarity had decreased from 0.7373482646933146 to 0.5733398760974173 for the sames line
the two lines are:
corporate code 123-234 you ca also tap on this link to verify corporate.co/1234
corporate code 134-456 you ca also tap on this link to verify corporate.co/5667
and similarity is 0.5733398760974173
- Is there any solution to avoid decreasing in similarity between sentences when their similar line messages increase in the input? (tfidf could be the problem here? when similar sentence number increases similarity decreases due to tfidf?)
- is there any solution to cluster similar messages?
i.e the input above, contains multiple sentences like:
hello world 123
same for the sentences like:
corporate code 123-234 you ca also tap on this link to verify corporate.co/1234
could they be grouped based on similarities output?