How to integrate on premise logs with GCP stackdriver

user_1047897 0
Viewed 2892

I am evaluating stackdriver from GCP for logging across multiple micro services. Some of these services are deployed on premise and some of them are on AWS/GCP. Our services are either .NET or nodejs based apps and we are invested in winston for nodejs and nlog in .net.

I was looking @ integrating our on-premise nodejs application with stackdriver logging. Looking @ https://cloud.google.com/logging/docs/setup/nodejs the documentation it seems that there we need to install the agent for any machine other than the google compute instances. Is this correct?

if we need to install the agent then is there any way where I can test the logging during my development? The development environment is either a windows 10/mac.

google-cloud-stackdriver google-cloud-platform
3 Answers

There's a new option for ingesting logs (and metrics) with Stackdriver as most of the non-google environment agents look like they are being deprecated. https://cloud.google.com/stackdriver/docs/deprecations/third-party-apps

A Google post on logging on-prem resources with stackdriver and Blue Medora https://cloud.google.com/solutions/logging-on-premises-resources-with-stackdriver-and-blue-medora

for logs you still need to install an agent on each box to collect the logs, it's a BindPlane agent not a Google agent.

user_118863580

For node.js, you can use the @google-cloud/logging-winston and @google-cloud/logging-bunyan modules from anywhere (on-prem, AWS, GCP, etc.). You will need to provide projectId and auth credentials manually if not running on GCP. Instructions on how to set these up is available in the linked pages.

When running on GCP we figure out the exact environment (App Engine, Compute Engine, etc.) automatically and the logs should up under those resources in the Logging UI. If you are going to use the modules from your development machines, we will report the logs against the 'global' resource by default. You can customize this by passing a specific resource descriptor yourself.

Let us know if you run into any trouble.

user_50518180

I tried setting this up on my local k8s cluster. By following this: https://kubernetes.io/docs/tasks/debug-application-cluster/logging-stackdriver/

But i couldnt get it to work, the fluentd-gcp-v2.0-qhqzt keeps crashing.

Also, the page mentions that there are multiple issues with stackdriver logging if you DONT use it on google GKE. See the screenshot.

I think google is trying to lock you in into GKE.

enter image description here

user_4733580
Related