HowTo do Cassandra insert in multi-thread?

Question

I read a big file to insert to Cassandra. I tried to create a Cassandra session, and pass it to multiple thread. I wish all thread use same session, to avoid thousands times reconnection.

I found the pool never call insert_chunk function because not see logging. Any idea why the session can't be passed to in a thread? I also tried to make session a global instead of parameter, the insert_chunk be called, but stuck at session.prepare

def insert_chunk(chunk, session):
    logging.info("insert chunck started")
    prepared = session.prepare(...)
    for line in chunck:
       session.execute(prepared, values)


main():
        cluster = Cluster()
        session = cluster.connect()
        with open("BIG_JSON", mode="r") as _file:
            objects = ijson.items(_file, "item")
            with Pool(20) as pool:
                while True:
                    chunk = list(itertools.islice(objects, 10000))
                    if len(chunk) <= 0:
                        break
                    pool.apply_async(
                        insert_chunk,[chunk, session],
                    )
                pool.close()
                pool.join()

Alex_Ott_18627 · Answer

Driver documentation says:

All of the patterns discussed above may be used over multiple processes using the multiprocessing module. Multiple processes will scale better than multiple threads, so if high throughput is your goal, consider this option. Be sure to never share any Cluster, Session, or ResponseFuture objects across multiple processes. These objects should all be created after forking the process, not before.

and links to the blog post that describe how to do it correctly.

But if everything you need is to load the data from a giant JSON file, then I have another suggestion - use DSBulk tool from DataStax (works with both DSE & Cassandra). This tool can read data from JSON, heavily optimized for loading & unloading of the data, and very flexible about mapping data from JSON into a table.

There is a series of blog posts on its usage:

HowTo do Cassandra insert in multi-thread?

1 Answers