Fluentd Configuration File, get filename in <source> and pass it to <match> as a tag

user_4917500 0
Viewed 2363

I am trying to write a clean configuration file for fluentd + fluentd-s3-plugin and use it for many files.

I want to avoid copy and pasting every <source> and every <match> for every file, so I would like to make it kinda dynamic. What I have until now:

  <source>
    @type tail
    path /var/www/blabla/blabla/log/production.log
    pos_file /var/www/blabla/blabla/log/production.log.pos
    tag production-log
    format /(?<time>.*)/
  </source>


  <match production-log>
    @type s3

      s3_bucket xxxx
      s3_region xxxx

      path "staging/%Y/%m/%d/#{Socket.gethostname}/"
      s3_object_key_format "%{path}productionlog-%{time_slice}-#{Socket.gethostname}-%{index}.%{file_extension}"

      # if you want to use ${tag} or %Y/%m/%d/ like syntax in path / s3_object_key_format,
      # need to specify tag for ${tag} and time for %Y/%m/%d in <buffer> argument.
      <buffer tag,time>
        @type file
        tag ${tag}
        path /var/www/blabla/blabla/log/buffer/
        timekey 3600 # (default 3600) 1 hour partition
        timekey_wait 1m
        timekey_use_utc true # use utc

      </buffer>
      <format>
        @type json
      </format>
  </match>

As you can see, I use production-log many times, so my questions are 2:

How do I use the filename in tag?

How do I use this passed tag in s3_object_key_format? (I know there are instructions commented but I don't get it)

fluentd config
1 Answers

Were you able to resolve above issue?. You do need to pass tag field to buffer section but no need to use inside buffer section.

Here is working configuration for me. Hope this helps, if it's not resolved already. My different sources send different value for tag field and under S3, I create separate folder by tag field and put objects under them.

So it will be like(app1 and app2 are tags)

mybucket-->topPrefix-->app1-->app.log

mybucket-->topPrefix-->app2-->app.log

<match>
  @type s3

  s3_bucket "mybucket"
  path "topPrefix/${tag}"
  s3_object_key_format "%{path}/%{time_slice}_#{Socket.gethostname}_%{index}.%{file_extension}"
  include_time_key true
  time_format "%Y-%m-%dT%H:%M:%S.%L"

  <buffer tag,time>
    @type file
    path /fluentd/buffer/s3
    timekey_wait 1m
    timekey 1m
    chunk_limit_size 64m
  </buffer>
  <format>
    @type json
  </format>
  time_slice_format %Y%m%d%H%M
</match>
user_50307090
Related