Neo4j: dependence of execution speed on batch size of input parameters

Viewed 41

I'm using Neo4J to identify the connections between different node labels.

Neo4J 4.4.4 Community Edition

DB rolled out in docker container with k8s orchestrating.

MATCH (source_node: Person) WHERE source_node.name in $inputs
MATCH (source_node)-[r]->(child_id:InternalId)
WHERE r.valid_from <= datetime($actualdate) < r.valid_to
WITH [type(r), toString(date(r.valid_from)), child_id.id] as child_path, child_id, false as filtered
CALL apoc.do.when(filtered,
'RETURN child_path as full_path, NULL as issuer_id',
'OPTIONAL MATCH p_path = (child_id)-[:HAS_PARENT_ID*0..50]->(parent_id:InternalId)
    WHERE all(a in relationships(p_path) WHERE a.valid_from <= datetime($actualdate) < a.valid_to) AND 
        NOT EXISTS{ MATCH (parent_id)-[q:HAS_PARENT_ID]->() WHERE q.valid_from <= datetime($actualdate) < q.valid_to}
    WITH DISTINCT last(nodes(p_path)) as i_source,
    reduce(st = [], q IN relationships(p_path) | st + [type(q), toString(date(q.valid_from)), endNode(q).id])
    as parent_path, CASE WHEN length(p_path) = 0 THEN NULL ELSE parent_id END as parent_id, child_path

    OPTIONAL MATCH (i_source)-[r:HAS_ISSUER_ID]->(issuer_id:IssuerId)
    WHERE r.valid_from <= datetime($actualdate) < r.valid_to
    RETURN DISTINCT CASE issuer_id WHEN NULL THEN child_path + parent_path + [type(r), NULL, "NOT FOUND IN RELATION"]
    ELSE child_path + parent_path + [type(r), toString(date(r.valid_from)), toInteger(issuer_id.id)]
    END as full_path, issuer_id, CASE issuer_id WHEN NULL THEN true ELSE false END as filtered',
    {filtered: filtered, child_path: child_path, child_id: child_id, actualdate: $actualdate}
)
YIELD value 
RETURN value.full_path as full_path, value.issuer_id as issuer_id, value.filtered as filtered
  1. When query executing on a large number of incoming names (Person), it is processed quickly for example for 100,000 inputs it takes ~2.5 seconds. However, if 100,000 names are divided into small batches and fore each batch query is executed sequentially, the overall processing time increases dramatically: 100 names batch is ~2 min 1000 names batch is ~10 sec Could you please provide me a clue why exactly this is happening? And how I could get the same executions time as for the entire dataset regardless the batch size?

  2. Is the any possibility to divide transactions into multiple processes? I tried Python multiprocessing using Neo4j Driver. It works faster but still cannot achieve the target execution time of 2.5 sec for some reasons.

  3. Is it any possibility to keep entire graph into memory during the whole container lifecycle? Could it help resolve the issue with the execution speed on multiple batches instead the entire dataset?

Essentially, the goal is to use as small batches as possible in order to process the entire dataset.

Thank you.

PS: Any suggestions to improve the query are very welcome.)

1 Answers

You pass in a list - then it will use an index to efficiently filter down the results by passing the list to the index, and you do additional aggressive filtering on properties.

So if you run the query with PROFILE you will see how much data is loaded / touched at each step.

A single execution makes more efficient use of resources like heap and page-cache.

For individual batched executions it has to go through the whole machinery (driver, query-parsing, planning, runtime), depending if you execute your queries in parallel (do you?) or sequentially, the next query needs to wait until your previous one has finished.

Multiple executions also content for resources like memory, IO, network.

Python is also not the fastest driver esp. if you send/receive larger volumes of data, try one of the other languages if that serves you better.

Why don't you just always execute one large batch then?

With Neo4j EE (e.g. on Aura) or CE 5 you will also get better runtimes and execution.

Yes if you configure your page-cache large enough to hold the store, it will keep the graph in memory during the execution.

If you run PROFILE with your query you should also see page-cache faults, when it needs to fetch data from disk.

Related