Ingest email attachments on ElasticSearch

Question

I'm trying to use ELK pipeline to read an email (IMAP), extract generic attachments (mainly PDF, eventually doc or ppt) and put them on ElasticSearch.

This is what I was able to do:

Loading directly to ElasticSearch from file some base64 data using Logstash, using the Ingest Attachment Processor on ElasticSearch to read the base64 content.
Loading data from IMAP (exchange email) I can correctly load all email information on ElasticSearch except the attachment (what I need).

The first solution works fine and does what I am looking for, except that it doesn't extract attachments directly from the email and that I have hardcoded base64 data inside the files. With the second solution I have a field x-ms-has-attach: yes on Kibana, but there isn't anywhere the attachment itself. The imap plugin is intended to load only the content of the email without the attachment?

What am I missing? Could you suggest me a pipeline to achieve what I am looking for?

This is my logstash configuration for the first example:

input {
  file {
    path => "/my/path/to/data/*"
    start_position => "beginning"
#    sincedb_path => "/my/path/to/sincedb"
    sincedb_path => "/dev/null"
    close_older => 0
    tags => ["attachment"]
   }
}

output {
  elasticsearch {
    index     => "email-attachment"
    hosts     => [ "localhost:9200" ]
  }
}

This is the pipeline:

PUT _ingest/pipeline/email-attachment
{
  "description": "Pipeline to parse an email and its attachments",
  "processors": [
    {
      "attachment" : {
        "field" : "message"
      }
    },
    {
      "remove" : {
        "field" : "message"
      }
    },
    {
      "date_index_name" : {
        "field" : "@timestamp",
        "index_name_prefix" : "email-attachment-",
        "index_name_format": "yyyy-MM",
        "date_rounding" : "M"
      }
    }
  ]
}

This is my logstash configuration for the second example:

input {
  imap {
    host => "my.domain.it"
    password => "mypassword"
    user => "myuser"
    port => 12345
    type => "imap"
    secure => true
    strip_attachment => true
  }
}

output {
  elasticsearch {
    index     => "email-attachment"
    hosts     => [ "localhost:9200" ]
  }
}

UPDATE

I'm using version 5.2.2

Ingest email attachments on ElasticSearch

2 Answers