We have dataproc clusters.And I am trying to list the jobs in the cluster using a service account. I am able to list the jobs in a certain cluster but not on few others within the same project. No errors. But jobs are not showing up. I am using the code below. Please let me know for any thoughts.
from google.cloud import dataproc_v1 as dataproc
from google.cloud import bigquery
import google.cloud as resources
from google.oauth2 import service_account as sa
from google.cloud.dataproc_v1 import JobControllerClient
client = Clients.get_client(ClientType.DATAPROC, name="x")
client_cluster = Clients.get_client(ClientType.DATAPROC_CLUSTER, name="x")
clus_collec = []
try:
cl_seq = client_cluster.list_clusters(project_id='y',
region='us-central1')
for k in cl_seq:
clus_collec.append(k.cluster_name)
for m in clus_collec:
data = client.list_jobs(project_id='y', region='us-central1', filter=f"clusterName={m} AND labels.execution_date=2022-08-25")
print(data)
except Exception as e:
print(e)