Logs aren't arriving in Cloud Logging from Google Compute Engine

user_5959293 0
Viewed 180

I have a VM instance running in GCE (using the Container Optimised OS) and within that I have an actively running container that is generating json logs. I can see these logs when I navigate to /var/lib/docker/containers/<CONTAINER_IMAGE>/<CONTAINER_IMAGE>-json.log.

In the same Instance, another docker container is running using the image gcr.io/stackdriver-agents/stackdriver-logging-agent:1.8.4. This was automatically set up when I created the VM.

The VM has permission to access to Cloud Logging and the Cloud Logging API is enabled. I have also followed the steps here and added google-logging-enabled to the metadata with a value of true.

When the VM is started, the logging agent seems to spin up correctly and emits a log saying that it is tailing the log file of the docker container I want logs for, however the logs within that file never appear in Google Logging. Below is a screenshot of the logs that do make it to Cloud Logging:

enter image description here

enter image description here

I have had this issue for a while now so would be very grateful for any help with this issue! Thanks in advance (:

google-cloud-logging stackdriver google-compute-engine
2 Answers

In the json logs I was providing, the time format used was not being accepted by fluentd. I've been able to get around that by adding:

reserve_time true

to the filter in the default config. Now the config ignores any nested fields with time specified. I learned of this from here.

user_59592930

Google logging uses a fluentd to catch the logs.

You can reconfugure fluentd to include additional log files.

Create a file /etc/google-fluentd/config.d/my_app_name.conf and put in the file a line in a format path /path/to/my/log. Here are more examples in the fluentd documentation.

You can also specify how the file is going to be parsed: as a single string type field or in more structured way (more convinient when you're looking for something). Again - here's some more info about fluentd's output plugins.

Finally go ahead and read the fluentd documentation to have a better understanding on using this tool.

user_122572500
Related