GCP Monitoring agent increasing memory usage continuously

user_4868978 0
Viewed 1068

I have installed GCP monitoring and logging agent on my compute engine instance. It has increased memory consumption more than 50% from the time it was installed. enter image description here

Any way to stop the memory utilization and reset back to the initial memory consumption? I have 3.75 GB RAM, out of which more than 3 GB consumed and more than 2 GB is being consumed by this process "/opt/google-fluentd/embedded/bin/ruby -Eascii-8bit:ascii-8bit /usr/sbin/google-fluentd --log /var/log/google-fluentd/google-fluentd.log --daemon /var/run/google-fluentd/google-fluentd.pid --under-supervisor"

Update:

After restart google-fluentd service, it brings down memory usage. But need to know the reason of its increased memory consumption. Is it a bug in fluentd service? enter image description here

google-cloud-monitoring google-compute-engine google-cloud-platform
1 Answers

Yes, it seems to be a known issue. The Logging Agent product team is still working on finding a fix for this issue. You can track google-fluentd (monitoring service) memory usage increase for updates.

Meanwhile the only workaround to solve this is to schedule a cron job to restart the fluentd agent periodically.

To restart the agent periodically run the following command on your instance:

 $ sudo service google-fluentd restart

Another recommendation is to check that there are not multiple Logging agent instances running on the VM (periodically).

Use ps -aux | grep "/usr/sbin/google-fluentd" to show running agent processes (there should be only two: one supervisor and one worker), and sudo netstat -nltp | grep :24231 to show running processes that occupy the port. Kill older instances as seen fit.

Edit :

Check whether your fluent-plugin-systemd version is upgraded to 1.0.5 by using the command:

$ /opt/google-fluentd/embedded/bin/gem list | grep fluent-plugin-systemd

If it is not upgraded to 1.0.5, you can upgrade using fluent-plugin-systemd 1.0.5.

If you have fluent-plugin-systemd 1.0.5 but are still seeing the issue, it might be the buffer output plugin issue that is still under investigation in https://github.com/fluent/fluentd/issues/3401

user_157451530
Related