How to improve read performance of Azure Blob Storage using PyArrow

Question

I am trying to read a subset of the nyc taxi data from Azure Open Data Storage. I want to understand what's happening in the background of PyArrow and how I can improve its read performance. The data stored is in Parquet files which I believe supports random access and should avoid downloading more data than needed if say I wanted just a column or wanted to filter the data.

The end goal here is perform this task on another dataset (which I can't share here). However this dataset will be in the arrow format as opposed to Parquet. I have converted a subset of the nyc taxi data to the arrow format and tried these examples on my own Azure blob storage with no real improvement.

The examples below are with Dask and directly with PyArrow. As I understand they both use Arrow under the hood as well as adlfs.

Some observations:

This read takes around 2 minutes
I can monitor my network and see a significant about of data being downloaded (500 MB or so) which I can't understand why as the data actually returned is only around 40 MB in memory.
The Arrow dataset returned is a lot less memory (12.6 MB) as opposed to the Dask counterpart (37.7 MB), I am not sure why.
I only reach a max download speed of around 30 mbps when running these blocks. But I notice if I spin up a new concurrent notebook and run it as well I can increase my throughput. Which tells me there must be some way to increase the concurrency and I have tried doubling the PyArrow IO thread count from 8 to 16 with no difference being shown.

PyArrow Example

import adlfs
import pyarrow.dataset as ds
import pyarrow as pa

pa.set_io_thread_count(16)
print(pa.io_thread_count())

fs = adlfs.AzureBlobFileSystem(account_name='azureopendatastorage')

ds_f = ds.dataset('nyctlc/green/puYear=2019/', filesystem=fs, format='parquet')

df = (
    ds_f
    .scanner(
        columns={ # Selections and Projections
            'passengerCount': ds.field(('passengerCount')),
        },
    )
    .to_table()
    .to_pandas()
)

df.info()

Output:

16

RangeIndex: 3297967 entries, 0 to 3297966
Data columns (total 1 columns):
 #   Column          Dtype
---  ------          -----
 0   passengerCount  int32
dtypes: int32(1)
memory usage: 12.6 MB

Dask Example - The focus here is how do this correctly in PyArrow, I decided to try this in Dask to see if there was any difference.

import dask.dataframe as dd

storage_options = {'account_name': 'azureopendatastorage'}
ddf = dd.read_parquet(
    'az://nyctlc/green/puYear=2019/puMonth=*/*.parquet',
    columns=['passengerCount'],
    storage_options=storage_options,
    engine="pyarrow"
)

ddf.compute().info()

Output:


Int64Index: 3297967 entries, 0 to 0
Data columns (total 1 columns):
 #   Column          Dtype
---  ------          -----
 0   passengerCount  int32
dtypes: int32(1)
memory usage: 37.7 MB

Keen to understand what I am missing here as I would have thought the read would be quite quick given the amount of data being requested.

How to improve read performance of Azure Blob Storage using PyArrow

0 Answers