neo4j cypher performance with parameter as a list

Question

We are running neo4j version 3.3. we have 2 types of nodes (total of 10k nodes) which have a total of 650k relationships. We heap size of the memory is 8GB.
The Nodes have a UNIQUE CONSTRAINT on the node ID.
We are using the official neo4j-python official driver (we also tried to use the py2neo driver but the performance was even worse).
While running the following query, the performance is VERY problematic. For a 1 hop distance it takes a few minutes (even for a list of several nodes). For a 2 hop distance (like the query below) with a list of 1 node it takes over 40 mintues.
Any ideas how to improve the performance?

query = '''MATCH (n1:label1)
            WHERE n1.ID IN {list}            
            MATCH paths=((n1)-[:relType*..2]->(n2))             
            WHERE n1.ID <> n2.ID AND (n2:label1 OR n2:label2)
            RETURN DISTINCT paths 
            UNION
            MATCH (n1:label2)
            WHERE n1.ID IN {list}            
            MATCH paths=((n1)-[:relType*..2]->(n2))             
            WHERE n1.ID <> n2.ID AND (n2:label1 OR n2:label2)
            RETURN DISTINCT paths'''

with driver.session() as session:
    results = list(session.run(query, parameters={'list':list_nodes}))
    if results:
        df = neo4j_graph_to_df(results)

The function to process the result is below:

def neo4j_graph_to_df(paths):
    paths_dict=OrderedDict()
    print(paths)
    for (pathID, e) in enumerate(paths):
        paths_dict[pathID]=OrderedDict()
        nodes_list = [n for n in e['paths'].nodes] 
        rels_list = [r for r in e['paths'].relationships]
        pathl = [x for x in itertools.chain.from_iterable(itertools.zip_longest(nodes_list, rels_list)) if x ]
        for i, p in enumerate(pathl):          
            if isinstance(p, neo4j.v1.types.Node):            
                paths_dict[pathID]['Node'+str(i)+'Label']= str(next(iter(p.labels)))
                dicti = dict(zip(['Node'+str(i)+str(np) for np in p.properties.keys()], p.properties.values()))
                paths_dict[pathID] = OrderedDict( {**paths_dict[pathID], **dicti} )                      
            if isinstance(p, neo4j.v1.types.Relationship): 
                paths_dict[pathID]['Rel'+str(i-1)]=p.type
                dicti = dict(zip(['Rel'+str(i)+str(rp) for rp in p.properties.keys()], p.properties.values()))
                paths_dict[pathID] =OrderedDict( {**paths_dict[pathID], **dicti } )
    df = pd.DataFrame.from_dict(paths_dict, orient='index').fillna('0')
    df = df.drop_duplicates().reset_index()
    return df

user_974731 · Answer

You stated that you have "a UNIQUE CONSTRAINT on the the node ID", but you actually need 2 constraints (or indexes). Each node label (label1 and label2) needs its own constraint (or index) on the ID property. For example:
```
CREATE CONSTRAINT ON (lab1:label1) ASSERT lab1.ID IS UNIQUE;

CREATE CONSTRAINT ON (lab2:label2) ASSERT lab2.ID IS UNIQUE;
```

With the above constraints (or indexes), this query should be faster:

MATCH (n1:label1)
USING INDEX n1:label1(ID)
WHERE n1.ID IN {list}
WITH COLLECT(n1) AS ns1
MATCH (n2:label2)
USING INDEX n2:label2(ID)
WHERE n2.ID IN {list}
WITH ns1 + COLLECT(n2) AS ns
UNWIND ns AS n
OPTIONAL MATCH path1=(n)-[:relType*..2]->(n31:label1)
WHERE n.ID <> n31.ID
OPTIONAL MATCH path2=(n)-[:relType*..2]->(n32:label2)
WHERE n.ID <> n32.ID
WITH COLLECT(path1) + COLLECT(path2) AS paths
UNWIND paths AS path
RETURN DISTINCT path

It incorporates USING INDEX clauses to give the Cypher planner a hint that it should use indexing to quickly get the starting nodes of interest (because the planner may not do that automatically). It then uses 2 OPTIONAL MATCH clauses to look for just label1 and label2 end nodes.

[UPDATE]

Instead of #2 above, you could take advantage of one of the Path Expander APOC procedures, since many allow you to specify end node labels when generating paths.

For example:

MATCH (n1:label1)
WHERE n1.ID IN {list}
WITH COLLECT(n1) AS ns1
MATCH (n2:label2)
WHERE n2.ID IN {list}
WITH ns1 + COLLECT(n2) AS startNodes
CALL apoc.path.expandConfig(
  startNodes,
  {labelFilter: '>label1|>label2', minLevel: 1, maxLevel: 2}
) YIELD path
RETURN DISTINCT path;

user_6835358 · Answer

You can try creating an index in the n.ID property:

CREATE INDEX ON :label1(ID)
CREATE INDEX ON :label2(ID)

Note that indexes are created in background tasks by the database. Neo4j will automatically pick up and start using the index once it has been created and brought online.

Also, try changing your query to use only one MATCH instead of two:

MATCH paths=((n1:label1)-[:relType*..2]->(n2))             
WHERE n1.ID IN {list} AND n1.ID <> n2.ID AND (n2:label1 OR n2:label2)
RETURN DISTINCT paths 
UNION
MATCH paths=((n1:label2)-[:relType*..2]->(n2))             
WHERE n1.ID IN {list} AND n1.ID <> n2.ID AND (n2:label1 OR n2:label2)
RETURN DISTINCT paths

neo4j cypher performance with parameter as a list

2 Answers