What is a good practice to map my Elasticsearch index?

Viewed 132

In order to model different types of metrics and audit logs when a search results should include both of them.

What is more encouraged in terms of fast search performance and indexing efficiency from the 3 options below:

  1. Mapping different subfields for each object
  2. Using a generic object that has predefined fields and a dynamic extra metadata field
  3. Using a different index for each object

Examples for each option:

Option 1 - Mapping different subfields under the same index unified_index_00001

{
 'type': 'audit_log',
 'audit_log_object': {
     'name': 'object_name',
     'action': 'edit',
     'before': 'field_value', 
     'after': 'field_value_update'
     }
},
{
 'type': 'network',
 'network': {
      'name': 'network_name',
      'event': 'access',
      'ip': 'xxx.xxx.xxx.xxx'
   }
},
{...} ...
  

Option 2 - Mapping a generic object under the same index unified_index_00001

{
 'type': 'audit_log',
 'name': 'object_name',
 'action': 'edit'
 'meta': {
    'before': 'field_value', 
    'after': 'field_value_update'
 }
},
{
 'type': 'network',
 'name': 'network_name',
 'action': 'access',
 'meta': {
     'ip': 'xxx.xxx.xxx.xxx'
   }
},
{...} ...
  

Option 3 - Using a different index for each object
audit_log_index_00001

{
 'name': 'object_name',
 'action': 'edit',
 'before': 'field_value', 
 'after': 'field_value_update'
},
{...}
...

metric_index_00001

{
  'name': 'network_name',
  'event': 'event_type',
  'ip': 'xxx.xxx.xxx.xxx'
},
{...} ...
  

Note: There is only need for Term indexing (no need of text searches)

1 Answers

In Elasticsearch, you usually want to start with the queries and then work backwards from there.

If you always query only events of one type, separate indices make sense. If there are mixed queries - you would be happier with a joint index, and usually with extracted common fields ("option 2") otherwise those queries won't really work.

Also, take into account Elasticsearch limitations:

  • fields per index (1000 by default)
  • shards and indices per cluster (thousands but still)
  • etc
Related