Use case: Set the maximum number of messages (within a timeframe) to be sent to a target service.
Example. We collect logs from service X which has these kind of logs:
{"@timestamp":"2020-10-30T13:00:00.310Z","level":"INFO","message":"This is some event"}
{"@timestamp":"2020-10-30T13:00:00.315Z","level":"WARN","message":"This is warn abc123"}
{"@timestamp":"2020-10-30T13:00:00.325Z","level":"WARN","message":"This is warn abc123"}
{"@timestamp":"2020-10-30T13:00:00.327Z","level":"WARN","message":"This is warn abc123"}
{"@timestamp":"2020-10-30T13:00:00.335Z","level":"WARN","message":"This is warn xyz123"}
As you can see the same warning (abc123) was logged multiple time by the service within 12ms. What I want is to send only one from them.
So fluentD should forward these to the target service:
{"@timestamp":"2020-10-30T13:00:00.310Z","level":"INFO","message":"This is some event"}
{"@timestamp":"2020-10-30T13:00:00.315Z","level":"WARN","message":"This is warn abc123"}
{"@timestamp":"2020-10-30T13:00:00.335Z","level":"WARN","message":"This is warn xyz123"}
Which timestamp to use or to have a counter doesn't matter for me.
Is there a filter,plugin for this use case? Something like where I can set a regex rule for the messages(for deciding whether more messages should be considered as equal) and a timeframe?