Create index-patterns from console with Kibana 6.0 or 7+ (v7.0.1)

Viewed 10727

I recently upgraded my ElasticStack instance from 5.5 to 6.0, and it seems that some of the breaking changes of this version have harmed my pipeline. I had a script that, depending on the indices inside ElasticSearch, created index-patterns automatically for some groups of similar indices. The problem is that with the new mapping changes of the 6.0 version, I cannot add any new index-pattern from the console. This was the request I used and worked fine in 5.5:

curl -XPOST "http://localhost:9200/.kibana/index-pattern" -H 'Content-  Type: application/json' -d'
{
  "title" : "index_name",
  "timeFieldName" : "execution_time"
}'

This is the response I get now, in 6.0, from ElasticSearch:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Rejecting mapping update to [.kibana] as the final mapping would have more than 1 type: [index-pattern, doc]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Rejecting mapping update to [.kibana] as the final mapping would have more than 1 type: [index-pattern, doc]"
  },
  "status": 400
}

How could I add index-patterns from the console avoiding this multiple mapping issue?

5 Answers

The URL has been changed in version 6.0.0, here is the new URL:

http://localhost:9200/.kibana/doc/doc:index-pattern:my-index-pattern-name 

This CURL should work for you:

curl -XPOST "http://localhost:9200/.kibana/doc/index-pattern:my-index-pattern-name" -H 'Content-Type: application/json' -d'
{
  "type" : "index-pattern",
  "index-pattern" : {
    "title": "my-index-pattern-name*",
    "timeFieldName": "execution_time"
  }
}'

If you are Kibana 7.0.1 / 7+ then you can refer saved_objects API ex: Refer: https://www.elastic.co/guide/en/kibana/master/saved-objects-api.html (Look for Get, Create, Delete etc).

In this case, we'll use: https://www.elastic.co/guide/en/kibana/master/saved-objects-api-create.html

$ curl -X POST -u $user:$pass -H "Content-Type: application/json" -H "kbn-xsrf:true"  "${KIBANA_URL}/api/saved_objects/index-pattern/dummy_index_pattern" -d '{ "attributes": { "title":"index_name*", "timeFieldName":"sprint_start_date"}}'  -w "\n" | jq

and

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   327  100   250  100    77    543    167 --:--:-- --:--:-- --:--:--   543
{
  "type": "index-pattern",
  "id": "dummy_index_pattern",
  "attributes": {
    "title": "index_name*",
    "timeFieldName": "sprint_start_date"
  },
  "references": [],
  "migrationVersion": {
    "index-pattern": "6.5.0"
  },
  "updated_at": "2020-02-25T22:56:44.531Z",
  "version": "Wzg5NCwxNV0="
}

Where $KIBANA_URL was set to: http://my-elk-stack.devops.local:5601

If you don't have jq installed, remove | jq from the command (as listed above).

PS: When KIBANA's GUI is used to create an index-pattern, Kibana stores its i.e. index ID as an alpha-numeric value (ex: laskl32ukdflsdjflskadf-sdf-sdfsaldkjfhsdf-dsfasdf) which is hard to use/find/type when doing GET operation to find info about an existing index-pattern using the following curl command.

If you passed index pattern name (like we did above), then in Kibana/Elasticsearch, it'll story the Index-Pattern's ID by the name you gave to the REST call (ex: .../api/saved_objects/index-pattern/dummy_index_pattern")

here: dummy_index_pattern will become the ID (only visible if you hover over your mouse on the index-pattern name in Kibana GUI) and

it'll have it's index name as: index_name* (i.e. what's listed in GUI when you click on Kibana Home > Gear icon > Index Patterns and see the index patterns listed on the right side.

NOTE: The timeFieldName is very important. This is the field, which is used for looking for time-series events (i.e. especially TSVB Time Series Visual Builder Visualization type). By default, it uses @timestamp field, but if you recreate your index (instead of sending delta information to your target Elasticsearch index from a data source (ex: JIRA)) every time and send all data in one shot from scratch from a data source, then @timestamp won't help with Visualization's Time-Spanning/Window feature (where you change time from last 1 week to last 1 hour or last 6 months); in that case, you can set a different field i.e. sprint_start_date like I used (and now in Kibana Discover data page, if you select this index-pattern, it'll use sprint_start_date (type: date) field, for events.

To GET index pattern info about the newly created index-pattern, you can refer: https://www.elastic.co/guide/en/kibana/master/saved-objects-api-get.html --OR run the following where (the last value in the URL path is the ID value of the index pattern we created earlier:

curl -X GET "${KIBANA_URL}/api/saved_objects/index-pattern/dummy_index_pattern" | jq

or

otherwise (if you want to perform a GET on an index pattern which is created via Kibana's GUI/webpage under Page Index Pattern > Create Index Pattern, you'd have to enter something like this:

curl -X GET "${KIBANA_URL}/api/saved_objects/index-pattern/jqlaskl32ukdflsdjflskadf-sdf-sdfsaldkjfhsdf-dsfasdf" | jq 

For Kibana 7.7.0 with Open Distro security plugin (amazon/opendistro-for-elasticsearch-kibana:1.8.0 Docker image to be precise), this worked for me:

curl -X POST \
-u USERNAME:PASSWORD \
KIBANA_HOST/api/saved_objects/index-pattern \
-H "kbn-version: 7.7.0" \
-H "kbn-xsrf: true" \
-H "content-type: application/json; charset=utf-8" \
-d '{"attributes":{"title":"INDEX-PATTERN*","timeFieldName":"@timestamp","fields":"[]"}}'

Please note, that kbn-xsrf header is required, but it seems like it's useless as from security point of view.

Output was like:

{"type":"index-pattern","id":"UUID","attributes":{"title":"INDEX-PATTERN*","timeFieldName":"@timestamp","fields":"[]"},"references":[],"migrationVersion":{"index-pattern":"7.6.0"},"updated_at":"TIMESTAMP","version":"VERSION"}

I can't tell why migrationVersion.index-pattern is "7.6.0".

For other Kibana versions you should be able to:

  1. Open Kibana UI in browser
  2. Open Developers console, navigate to Network tab
  3. Create index pattern using UI
  4. Open POST request in the Developers console, take a look on URL and headers, than rewrite it to cURL

Create index-pattern in bulk with timestamp:

cat index_svc.txt
my-index1
my-index2
my-index3
my-index4
my-index5
my-index6

cat index_svc.txt | while read index; do
    echo -ne "create index-pattern ${index} \t"
    curl -XPOST "http://10.0.1.44:9200/.kibana/doc/index-pattern:${index}" -H 'Content-Type: application/json' -d "{\"type\":\"index-pattern\",\"index-pattern\":{\"title\":\"${index}2020*\",\"timeFieldName\":\"@timestamp\"}}"
    echo 
done
Related