import pyarrow not working <- error is "ValueError: The pyarrow library is not installed, please install pyarrow to use the to_arrow() function."

user_14330543 0
Viewed 15140

I have tried installing it in the terminal and in juypter lab and it says that it has been successfully installed but when I run df = query_job.to_dataframe() I keep getting the error " ValueError: The pyarrow library is not installed, please install pyarrow to use the to_arrow() function.". I have no idea how to fix this. Any advice? I am trying to access data from google data studio ultimately with the code,

from google.cloud import bigquery
import pandas
import numpy
import pyarrow
bigquery_client = bigquery.Client()
import os 
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] ='full file path here.json'
QUERY = """
SELECT * 
FROM `warehouse`
LIMIT 100
"""
query_job = bigquery_client.query(QUERY)
df = query_job.to_dataframe()
jupyter pyarrow google-bigquery
6 Answers

I got the same error message ModuleNotFoundError: No module named 'pyarrow' when testing your Python code. This behavior disappeared after installing the pyarrow dependency with pip install pyarrow.

Edit: It worked for me once I restarted the kernel after running pip install pyarrow

user_117342840
user_149970200

I had the same issue because I had pyarrow 2.0, however you will need version 1.0.1 . Try running this line: pip install pandas-gbq==0.14.0

user_113809770

To avoid using fetch_pandas_all(), I have used fetchall, and then converted result to pandas DataFrame I have used:

requirements.txt

snowflake-connector-python==2.4.3
pandas==1.2.4

dag.py

    def execute(self, **kwags):
        """
        :param kwargs: optional parameter. Can be used to provide task input context
        :return: returns query result in json format
        """

        ctx = snowflake.connector.connect(
            user=self.SNOWFLAKE_USER,
            password=self.SNOWFLAKE_PASSWORD,
            account=self.SNOWFLAKE_ACCOUNT
        )
        cs = ctx.cursor()
        try:
            cs.execute(self.sql_query)
            data = cs.fetchall()
            df = pd.DataFrame(data)
            print(f'\nQUERY RESULT: \n' \
                      f' {tabulate(df, headers="keys", tablefmt="psql", showindex="always")} \n')
        finally:
            cs.close()
        ctx.close()
        logging.info("Query executed successfully")
        return json.loads(data)
user_157933810

I have experienced a similar problem but then I used the pandas Dataframe method:

client = bigquery.Client()
try:
    df = client.query(query)
    df = pd.Dataframe(df)
except ValueError:
    print("google services not available or invalid credentials.")

df.head()
user_145990000

just need to install pyarrow using pip

df = client.query(query1).to_dataframe()
 data = df.to_json()
        
 print(data['total_transactions'][0])
 print(data['total_visits'][0])
user_55556450
Related