Doing the equivalent of log_struct in python logger

Current solution:

Update 1 - user Seth Nickell improved my proposed solution, so I update this answer as his method is superior. The following is based on his response on GitHub:

https://github.com/snickell/google_structlog

pip install google-structlog

Used like so:

import google_structlog

google_structlog.setup(log_name="here-is-mylilapp")

# Now you can use structlog to get searchable json details in stackdriver...
import structlog
logger = structlog.get_logger()
logger.error("Uhoh, something bad did", moreinfo="it was bad", years_back_luck=5)

# Of course, you can still use plain ol' logging stdlib to get { "message": ... } objects
import logging
logger = logging.getLogger("yoyo")
logger.error("Regular logging calls will work happily too")

# Now you can search stackdriver with the query:
# logName: 'here-is-mylilapp'

Original answer:

Based on an answer from this GitHub thread, I use the following bodge to log custom objects as info payload. It derives more from the original _Worker.enqueue and supports passing custom fields.

from google.cloud.logging import _helpers
from google.cloud.logging.handlers.transports.background_thread import _Worker

def my_enqueue(self, record, message, resource=None, labels=None, trace=None, span_id=None):
    queue_entry = {
        "info": {"message": message, "python_logger": record.name},
        "severity": _helpers._normalize_severity(record.levelno),
        "resource": resource,
        "labels": labels,
        "trace": trace,
        "span_id": span_id,
        "timestamp": datetime.datetime.utcfromtimestamp(record.created),
    }

    if 'custom_fields' in record:
        entry['info']['custom_fields'] = record.custom_fields

    self._queue.put_nowait(queue_entry)

_Worker.enqueue = my_enqueue

Then

import logging
from google.cloud import logging as google_logging

logger = logging.getLogger('my_log_client')
logger.addHandler(CloudLoggingHandler(google_logging.Client(), 'my_log_client'))

logger.info('hello', extra={'custom_fields':{'foo': 1, 'bar':{'tzar':3}}})

Resulting in:

Which then makes it much easier to filter according to these custom_fields.

Let's admit this is not good programming, though until this functionality is officially supported there doesn't seem to be much else that can be done.

This is not currently possible, see this open feature request on google-cloud-python for more details.

Official docs: Setting Up Cloud Logging for Python

You can write logs to Logging from Python applications

1. by using the Python logging handler included with the Logging client library, or
2. by using Cloud Logging API Cloud client library for Python directly.

I did not get the Python logging module to export jsonPayload, but the cloud logging library for Python works:

google-cloud-logging >= v.3.0.0 can do it

No need for Python logging workaround anymore, the only thing you need is to install Python >= 3.6 and

pip install google-cloud-logging

Then, you can use Python logging with

import os

# have the environment variable ready:
# GOOGLE_APPLICATION_CREDENTIALS
# Then:
GOOGLE_APPLICATION_CREDENTIALS = os.environ.get("GOOGLE_APPLICATION_CREDENTIALS")
client = google.cloud.logging.Client.from_service_account_json(
    GOOGLE_APPLICATION_CREDENTIALS
)
log_name = "test"
gcloud_logger = client.logger(log_name)

# jsonPayloads
gcloud_logger.log_struct(entry)
# messages
gcloud_logger.log_text('hello')
# generic (can find out whether it is jsonPayload or message)!
gcloud_logger.log(entry or 'hello')

You run these commands in a Python file outside of GCP and reach GCP with more or less a one-liner, you only need the credentials.

You can use the gcloud logger even for printing and logging in one go, taken from Writing structured logs, untested.

Python logging to log jsonPayload into GCP logs (TL/DR)

I could not get this to run!

You can also use the built-in logging module of Python with the workaround mentioned in the other answer, but I did not get it to run. It will not work if you pass a dictionary or its json.dumps() directly as a parameter, since then, you get a string output of the whole dictionary which you cannot read as a json tree. But it also did not work for me when I used the logger.info() to log the jsonPayload / json.dumps in an example parameter called extras.

import json
import os

#...

# https://googleapis.dev/python/logging/latest/stdlib-usage.html
GOOGLE_APPLICATION_CREDENTIALS = os.environ.get("GOOGLE_APPLICATION_CREDENTIALS")
client = google.cloud.logging.Client.from_service_account_json(
    GOOGLE_APPLICATION_CREDENTIALS
)
log_name = "test"
handler = CloudLoggingHandler(client, name=log_name)
setup_logging(handler)
logger = logging.getLogger()
logger.setLevel(logging.INFO)  # Set default level.

#...

    # Complete a structured log entry.
    entry = dict(
        severity="NOTICE",
        message="This is the default display field.",
        # Log viewer accesses 'component' as jsonPayload.component'.
        component="arbitrary-property"
    )

    # Python logging to log jsonPayload into GCP logs
    logger.info('hello', extras=json.dumps(entry))

I also tried the google-structlog solution of the other answer, that only threw the error:

google_structlog.setup(log_name=log_name) TypeError: setup() got an unexpected keyword argument 'log_name'

I used Python v3.10.2 and

google-auth==1.35.0
google-cloud-core==1.7.2
google-cloud-logging==1.15.0
googleapis-common-protos==1.54.0

Research steps gcloud logging (TL/DR)

Following the fixed issue (see the merge and close in the end) on github at googleapis / python-logging: Logging: support sending structured logs to stackdriver via stdlib 'logging'. #13

you find feat!: support json logs #316 :

This PR adds full support for JSON logs, along with standard text logs. Now, users can call logging.error({'a':'b'}), and they will get a JsonPayload in Cloud Logging, Or call logging.error('test') to receive a TextPayload

As part of this change, I added a generic logger.log() function, which serves as a generic entry-point instead of logger.log_text or logger.log_struct. It will infer which log function is meant based on the input type

Previously, the library would attach the python logger name as part of the JSON payload for each log. Now, that information will be attached as a label instead, giving users full control of the log payload fields

Fixes #186, #263, #13

With the main new version listing the new feature:

chore(main): release 3.0.0 #473