Neo4j word graph query

user_7412939 0
Viewed 425

I've created a Neo4j graph of Questionnaire completions which may have 1 or more FreeTextResponse answers. Each FreeTextResponse has any number of Words in it.

Words are related to other Words in the graph with an IS_RELATED_TO relationship that has a similarityValue property denoting how similar the words are semantically.

Any number of FreeTextResponses could be linked to the Words.

Ultimately I would like to find clusters of related Questionnaire completions based on the similarity of the FreeTextResponse answers.

My first step would seem to be writing a query that traverses through the graph from a Questionnaire completion node, through the FreeTextResponse answers, through the words, and through any FreeTextResponse answers that those Words also exist in.

Then the same again, but doing it through two or more word nodes and calculating a weight based on their cumulative similarity value.

The graph structure looks like this:

enter image description here

I've tried a number of approaches, but am struggling with the number of variables there are at play.

My best attempt at the first step is this:

match (qr:Questionnaire{QuestionnaireReturnID:186406})<-[:IN]-(ftr:FreeTextResponse)<-[:IN]-(w1:WordGraph_Word)-[:IN]->(ftr2:FreeTextResponse)-[:IN]->(qr2:Questionnaire)
WITH qr2, ftr, count(distinct w1) as frequency
WITH distinct qr2, length(split((ftr.fullSentenceString), ' ')) as wordCount, frequency
WITH
COLLECT({
QuestionnaireReturnID: qr2.QuestionnaireReturnID,
frequency: frequency,
wordCount: wordCount,
percentageMatch: (toFloat(frequency) / toFloat(wordCount)) * 100
}) as associatedQuestionnaires
UNWIND(associatedQuestionnaires) as a
WITH a order by a.percentageMatch desc where a.percentageMatch > 15
return a

...but I'm not sure that I trust the results and I don't think they take into account that there could be any number of FreeTextResponses for a Questionnaire.

What is the best approach for this kind of problem please?

Thank you!

Update:

I think I may have just worked out step 1, linking the directly linked Questionnaires as follows:

match (qr:Questionnaire{QuestionnaireReturnID:186406})<-[:IN]-(ftr:FreeTextResponse)<-[:IN]-(w:WordGraph_Word)
WITH COLLECT(w) as wordList, count(w) as allWordsInSourceCount
UNWIND(wordList) as words
WITH words, allWordsInSourceCount
MATCH (words)-[relIn:IN]->(ftr2:FreeTextResponse)-[:IN]->(qr2:Questionnaire) WHERE qr2.QuestionnaireReturnID <> 186406
WITH qr2, count(distinct words) as matchFrequency, allWordsInSourceCount
WITH qr2, matchFrequency, allWordsInSourceCount, (toFloat(matchFrequency) / toFloat(allWordsInSourceCount)) * 100 as percentageMatch
WITH qr2, matchFrequency, allWordsInSourceCount, percentageMatch order by percentageMatch desc where percentageMatch > 30
RETURN qr2.QuestionnaireReturnID, matchFrequency, allWordsInSourceCount, percentageMatch

cypher neo4j graph
1 Answers

It seems you are on the good track with your query. However I want to mention a couple of things regarding similarity of text and point you to a couple of links that might be useful for further learning :

TF-IDF

Counting similar word occurrences will generally lead to poor results. In the information retrieval field, a common technique for measuring similarity based on words is TF/IDF :

The measure is simple, TF is the term frequency in a particular document. There are multiple variations but let's take the simple one, raw term frequency for the given sentence : Today I went to the shopping center and then went back to home

The TF for the word shopping is one, while for to it is 2.

The Inverse Document Frequency (IDF) defines what is the importance of that word :

idf(word, document) = log ( total number of documents / number of documents containing that word )

The TF-IDF is then calculated as 

tf-idf(word, document, corpus) = tf(word, doc) . idf(word, corpus)

So, a word in a particular document will have higher tf-idf if the tf is high and the idf is low.

Now take back your use case, when comparing two FreeTextResponse, you can calculate also how much the tf-idf of the common words are closed to each other:

FTR => FreeTextResponse

similarity(ftr1, ftr2, wordA) = 1 - ( tf-idf(wordA, ftr1) - tf-idf(wordA, ftr2) )

Small explanation in English : if I have a document where dog appears 12 times and another document where it appears 1 time, then those two documents are maybe not speaking about the same thing.

Note that we generally use a variant of TF/IDF taking the document length into account, more info in the Wikipedia link.

Stopwords

Secondly, not all words are meaningful. This is what we call stopwords and they can be discarded during the similarity matching or even not be inserted in the graph at all. Those words are for eg : the, it, a,...

If you take as an example the Lucene search engine ( which is the base of Elastic and Solr ), you can find the default stopwords list for English in the analyzer :

static {
    final List<String> stopWords = Arrays.asList(
      "a", "an", "and", "are", "as", "at", "be", "but", "by",
      "for", "if", "in", "into", "is", "it",
      "no", "not", "of", "on", "or", "such",
      "that", "the", "their", "then", "there", "these",
      "they", "this", "to", "was", "will", "with"
    );
      ....

Similar Words

In your diagram, you show words related to each other by a SIMILAR_TO relationship with a similarity score.

Those can also be used for finding similarity, of course, you should take into account the depth in order to decrease the similarity.

Conclusion

Overall, I would not focus on making everything in one single Cypher query. I would really focus on trying to find what makes two documents similar in your specific domain, and then combine the results of multiple Cypher queries with your favourite programming language.

You can also look at some Neo4j extensions that might ease and advance your text analytics as well, like https://github.com/graphaware/neo4j-nlp

user_26623550
Related