Docker & gcplogs : message as json

Question

I run a project deployed with docker-compose using gcplogs driver. I wonder if it's possible to transform a message string written in stdout to a complex JSON payload.

Actually I have to use the following specific log format to retrieve logs in stackdriver, so the payload look like this in Stackdriver:

{
  "container": {…}   
  "instance": {…}   
  "message":  "service:php type:NOTICE message:\"The message I want to log\""   
 }

I've tried to format the message as JSON but ends up with the message property containing the stringified JSON:

{
  "container": {…}   
  "instance": {…}   
  "message":  "{\"service\":\"php\",\"type\":\"NOTICE\",\"message\":\"The message I want to log\"}"
 }

Is there any config in docker-compose to automatically parse the message string as the JSON payload? Like the option --payload-type=json when using gcloud with CLI? Maybe its possible with fluentd driver but is it correctly handled by gcp ?

Thanks folks!

user_2263517 · Answer

You can use the Google's version of fluentd which already comes preconfigured to work with Stackdriver (e.g. it auto-discovers the credentials when running from inside a GCP VM).

Install google-fluentd directly on your VM (outside of Docker)

$ curl -sSO https://dl.google.com/cloudagents/install-logging-agent.sh
$ sudo bash install-logging-agent.sh

You can remove all the default boilerplate configuration from fluentd config file (etc/google-fluentd/google-fluentd.conf on Ubuntu) and just add the configuration for your particular service (providing you don't need to use google-fluentd for anything else). In order to do so remove everything before the section starting with and insert the following configuration instead:
```
  @type forward
  port 24225
  bind 127.0.0.1



  @type add_insert_ids



  @type parser
  key_name message
  
    @type json
  
```
Reload fluentd:

$ sudo systemctl reload google-fluentd
Set fluentd as a logging driver with destination localhost:24225 for your app container, or simply change the default logging driver for the whole Docker Engine in /etc/docker/daemon.json:
```
{
  "log-driver": "fluentd",
  "log-opts": {
    "fluentd-address": "localhost:24225"
  }
}
```
and reload Docker:

$ sudo systemctl reload docker

From now on you should see your application logs nicely parsed out in the Stackdriver Logging viewer. If you don't, check the fluentd log (/var/log/google-fluentd/google-fluentd.log on Ubuntu) for possible issues with JSON parsing.

Docker & gcplogs : message as json

1 Answers