I have a few Google Cloud Big Query Tables of varying sizes like betweem 15GB to 150GB, which I want to analyse/create calculations on/build models, using Python.
All of this, ideally should be done within Google Cloud Environment. I have tried using Jupyter Notebook instance within Vertex AI, but the underlying VM instance's configuration is unable to read this amount of data (150GB) even after increasing its RAM to 360GB.
I am getting the following error:
Forbidden: 403 Response too large to return. Consider specifying a destination table in your job configuration. For more details, see https://cloud.google.com/bigquery/troubleshooting-errors
Kindly advice on possible solutions within vertex AI or should I use some other product of Google Cloud?