BigQuery client with multiprocessing.Pool

Viewed 159

From google docs:

Because the BigQuery client uses the third-party requests library by default and the BigQuery-Storage client uses grpcio library, both are safe to share instances across threads. In multiprocessing scenarios, the best practice is to create client instances after multiprocessing.Pool or multiprocessing.Process invokes os.fork().

On the one hand "safe to share instances across threads" is mentioned, on the other, to create client instances after "os.fork()" is done.
And I don't understand if I can create one client and pass to threads to use, or if each thread has to initialize client itself separately.

I realize multithreading with the following code:

with multiprocessing.Pool(processes=NUMBER_OF_PROCESSES) as proc:
    proc.map(my_bq_func, arguments_list)
0 Answers
Related