I have a simple BigQuery command that I invoke from Python client. Import to GS works fine , but when I import to S3 , I get a 400 error "google.api_core.exceptions.BadRequest: 400 Unknown keyword EXPORT"
I have seen sample from Google to load directly from BQ to S3 .. https://cloud.google.com/bigquery/docs/samples/bigquery-omni-export-query-result-to-s3 Is this only supported in BigQuery-OMNI ??
not sure what is missing ... please advise ...
There is no permission issue anywhere, I have double verified
from google.cloud import bigquery
from google.oauth2 import service_account
credentials = service_account.Credentials.from_service_account_file("dev-key.json", scopes=["https://www.googleapis.com/auth/cloud-platform"],)
client = bigquery.Client(credentials=credentials, project=credentials.project_id,)
bq_export_to_s3 = """
EXPORT DATA WITH CONNECTION `aws-us-east-1.s3-write-conn`
OPTIONS(
uri='s3://my-bucket/logs/edo/dengg_audit/bq-demo/temp4/*',
format='CSV',
overwrite=true,
header=false,
field_delimiter='^') AS
select col1 , col2 from `project.schema.table` where clientguid = '1234' limit 10
"""
bq_export_to_gs = """
EXPORT DATA OPTIONS(
uri='gs://my-bucket/logs/edo/dengg_audit/bq-demo/temp4/*',
format='CSV',
overwrite=true,
header=false,
field_delimiter='^') AS
select col1 , col2 from `project.schema.table` where clientguid = '1234' limit 10
"""
query_job= client.query(bq_export_to_s3)
results = query_job.result()
for row in results:
print(row)
pip3 install google-api-python-client==1.12.8 \
&& pip3 install google-cloud-storage==1.33.0 \
&& pip3 install google-cloud-bigquery==2.10.0
Error Message
