How should Pubsub, acting a log sink, fire a function without sending the log?

Question

I have been using this example of creating a Vertex AI monitoring job. It sends an email and have adapted it to send a Pubsub message, with @Jose Gutierrez Paliza's help.

I have got this working, sort of. But what seems to be happening is that Pubsub pushes the log to a function which errors.

My log sink includes:

When I look at logs I see an INFO entry:

   my-fn an_id Event data: {"insertId":"another_id...

followed by a separate ERROR entry:

...
ValueError: The pipeline parameter insertId is not found in the pipeline job input definitions.

So I assume Pubsub is sending the log to the function which gets extraneous crap, including insertId.

I can run the pipeline fine via Jupyter:

from google.cloud import pubsub

publish_client = pubsub.PublisherClient()
topic = f'projects/{PROJECT}/topics/{PUBSUB_TOPIC}'
data = {}
message = json.dumps(data)

_ = publish_client.publish(topic, message.encode())

So how do I the equivalent via Pubsub?

user_1773592 · Answer

It was the log being used (as data) in the following:

parameter_values = json.dumps(data).encode()

I set:

parameter_values={"project": project, "display_name": "some_name"}

How should Pubsub, acting a log sink, fire a function without sending the log?

1 Answers