I am trying to create a Tensorflow BigQuery ReadSession with 32 streams.
read_session = tensorflow_io_bigquery_client.read_session(
...
requested_streams=32,
data_format=BigQueryClient.DataFormat.AVRO
)
However, the number of streams created is limited to 7.
print(len(read_session.streams))
# -> 7
The documentation says the following:
"""
requested_streams: Initial number of streams. If unset or 0, we will
provide a value of streams so as to produce reasonable throughput.
Must be non-negative. The number of streams may be lower than the
requested number, depending on the amount parallelism that is reasonable
for the table and the maximum amount of parallelism allowed by the
system.
"""
So, 2 causes possible here:
1 - The number of streams is limited by the maximum amount of parallelism allowed by the system:
- I tried to choose a machine with 4x more CPUs, but I encountered exactly the same behavior. So it does not seems to be limited by the system.
2 - The number of streams is limited by the amount of parallelism that is reasonable for the table:
- I definitely do not understand what does this means. Is there someone able to explain to me how
the amount of parallelism that is reasonable for the tableis calculated? And how to increase it?