Python: parallel Neo4J batch writes

Question

I want to display word relationships in Neo4j 4.0. I wrote a scraper in Python 3.8 which reads 3.000 text files and afterwards stores the information in Neo4J with the Neo4j Python Driver 4.1. The whole process takes 0.5-2 seconds per text file. I am currently trying to speed up these writes by executing them parallel. After reading the docs:

sessions may only host one transaction at a time. For parallel execution, multiple sessions should be used. https://neo4j.com/docs/driver-manual/current/cypher-workflow/#driver-sessions

I decided to batch the processing of each file/text into one transaction in one session. So for each file I create and afterwards close a new session:

driver = GraphDatabase.driver('uri', auth=('user', 'password'), encrypted=False)

def feed_word_list(tx, word_list, title, headline):
    ...
    #example batch
    batch = {'batch':[{'from':'start','to':'end','from_tf':3,'to_tf':4}]}
    tx.run( 'UNWIND $batch as tuples '
            'MERGE (word1: Word {title: tuples.from}) '
            'MERGE (word2: Word {title: tuples.to}) '
            'MERGE (text1: Text {title: $title, headline: $headline}) '
            'MERGE (text1) -[:CONTAINS {term_frequency: tuples.from_tf}]-> (word1) '
            'MERGE (text1) -[:CONTAINS {term_frequency: tuples.to_tf}]-> (word2) '
            'MERGE (word1) -[:IS_NEIGHBORING]-> (word2) '
            , batch, title=title, headline=headline)

def parse_write(f):
    text = parse(f)
    with driver.session() as session:
        session.write_transaction(feed_word_list, text.text, text.identifier, text.headline)

#path is a directory of files
def scrape(path):
    #files is a list of files, which can processed independently 
    files = create_files_list(path)
    with Pool(2) as p:
        p.map(process_file, files)
    driver.close()

It runs into the following exception/error:

Transaction failed and will be retried in 1.0903621073189202s (LockClient[220] can't wait on resource RWLock[NODE(11398), hash=1487896506] since => LockClient[220] <-[:HELD_BY]- RWLock[NODE(10846), hash=1926906302] <-[:WAITING_FORsource RWLock[NODE(11398), hash=1487896506] since => LockClient[220] <-[:HELD_BY]- RWLock[NODE(10846), hash=1926906302] <-[:WAITING_FOR]- LockClient[222] <-[:HELD_BY]- RWLock[NODE(11398), hash=1487896506])
FIXME: should always disconnect before connect

My assumption would be that there too many parallel connections which are locking each other. But as far as I understood: with Pool(2) as p: there are only two parallel executions. So my questions is: How are parallel writes with python and Neo4j possible?

Python: parallel Neo4J batch writes

0 Answers