GCP alert if Docker container exits

Yes. You have to configure docker with Cloud logging driver, create a log based metric and an alerting policy on that metric.

Solution:

Configure docker with cloud logging driver.

• SSH to your instance.

• Run this command as root: dockerd --log-driver=gcplogs. This will forward your docker logs to Google Cloud Logging.

• If running a container-optimized OS then follow these steps:

  echo '{"log-driver":"gcplogs"}' | sudo tee /etc/docker/daemon.json

  sudo systemctl restart docker

• Try starting and exiting a container.

These are the logs generated whenever we exit a container. Keep in mind the two exit messages

Create a log based metric

• Go to Logging -> Select Log-based Metrics.

• Click on Create Metric.

• Metric Type: Counter. In Details, enter a Log Metric name (e.g. mysite-container-exited)

• Under Filter Selection -> Build filter, copy the following code

  resource.type="gce_instance"

  log_name="projects/myproject-12345/logs/gcplogs-docker-driver" replace myproject-12345 with your project name.

  jsonPayload.container.name="/mysite" change mysite to your container name.

  jsonPayload.message=~"exit$" This is regex that matches exit as the last word in a line.

It should look something like this.

Create an alerting policy

• Go to Monitoring -> Select Alerting.
• Click on Create Policy.
• Click on Select a Metric and search for your metric name (e.g. mysite-container-exited).
• Select your metric and click on Apply.
• Set Rolling window to 2 min and Rolling window function to count. Click Next.
• Set Alert Trigger to Any time series violates, Threshold postion to Above threshold and Threshold value to 1. Click Next.
• Select a notification Channel to send alerts to. If none exists then click on Manage Notification Channels and create one.
• Easiest one would be to add an email for notification channel.
• After creating go back, click on Refresh icon in Notification Channels and select it. Click ok.
• Click on Save Policy.

As we have seen that two exit commands are issued per container to the logs whenever we exit a container, thats's why the threshold is set to above 1.

You can monitor more containers by creating a new metric and changing the value of jsonPayload.container.name="/mysite" to your container name.

Now try to start and stop the container. You will receive an alert on email in 2-3 minutes.