How To Remove Quotation Marks (" ") In Geo Coordinates On Logstash Conf File

user_11265798 0
Viewed 285

I want to express a line on a map in Kibana and the map should be Geojson structure. The data that I have is a set of SQL table then I was about to transfer them to Elastic Search using Logstash like this

input{ ... }

filter{
    
 if [lat] and [lon] {
        mutate{convert =>  ["lat", "float"]}
        mutate{convert =>  ["lon", "float"]}
        mutate{convert =>  ["for_lat", "float"]}
        mutate{convert =>  ["for_lon", "float"]}

        mutate{
            add_field => {"[location-geotest][type]" => "multilinestring"}
            add_field => {"[location-geotest][coordinates]" => [["%{lon}", "%{lat}"]]}
            add_field => {"[location-geotest][coordinates]" => [["%{for_lon}", "%{for_lat}"]]}
            }
      }
}

However the logstash conf file failed to index the data on Elasticsearch

{
    :status=>400, 
    :action=>["index", {:_id=>"18022", :_index=>"geo_shape_test", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x687994b9>], 
    :response=> {
        "index"=>{
            "_index"=>"geo_shape_test", 
            "_type"=>"_doc", 
            "_id"=>"18022", 
            "status"=>400, 
            "error"=>{
                "type"=>"mapper_parsing_exception", 
                "reason"=>"failed to parse field [location-geotest] of type [geo_shape]", 
                "caused_by"=>{"type"=>"x_content_parse_exception", 
                    "reason"=>"[1:164] [geojson] failed to parse field [coordinates]", 
                    "caused_by"=>{
                        "type"=>"parse_exception", 
                        "reason"=>"geo coordinates must be numbers"
                    }
                }
            }
        }
    }
}

and this is the one of what logstash tried to index

{
                 "lat" => 37.567179953757886,
              "gps_id" => 10491,
           "timestamp" => 2020-11-22T06:10:45.000Z,
               "speed" => 17.25745240090587,
                 "lon" => 126.99598717854032,
             "for_lat" => 37.567179953757886,
          "@timestamp" => 2020-11-27T03:54:21.131Z,
             "for_lon" => 126.99598717854032,
            "@version" => "1",
    "location-geotest" => {
        "coordinates" => [
            [0] "[\"126.99598717854032\", \"37.567179953757886\"]",
            [1] "[\"126.99598717854032\", \"37.567179953757886\"]"
        ],
               "type" => "multilinestring"
    }
}

I think the problem is...

 "coordinates" => [
            [0] "[\"126.99598717854032\", \"37.567179953757886\"]",
            [1] "[\"126.99598717854032\", \"37.567179953757886\"]"
        ],

if I change the part, it will be...

 "coordinates" => [
            [0] [126.99598717854032, 37.567179953757886],
            [1] [126.99598717854032, 37.567179953757886]
        ],

But I can't find how to solve.

logstash kibana geojson elasticsearch json
1 Answers

I think the problem is as you say, the coordinates has to be float instead of strings. Apparently the mutate function converts the value back to string. As mentioned in

https://discuss.elastic.co/t/logstash-mutate-filter-always-stringifies-hash-and-array/25917

They suggest the solution to use a ruby script instead. This has been done for the linestring as.

https://discuss.elastic.co/t/geo-shape-geo-link-problems-with-coordinates/179924/4

From the data provided I don't see why you need multiline string? With only two points it should be enough to store as line string.

I tried it out with

filter{
    
 if [lat] and [lon] {

        mutate{
            convert =>  ["lat", "float"]
            convert =>  ["lon", "float"]
            convert =>  ["for_lat", "float"]
            convert =>  ["for_lon", "float"]
            
            add_field => {"[location-geotest][type]" => "linestring"}
        }

      ruby{
          code => "event.set('[location-geotest][coordinates]', [[event.get('lon'), event.get('lat')], [event.get('for_lon'), event.get('for_lat')]])"
      }
  }
}

and get the result:

    "location-geotest" => {
           "type" => "linestring",
           "coordinates" => [
                 [0] [
                      [0] 126.99598717854032,
                      [1] 37.567179953757886
                     ],
                 [1] [
                      [0] 126.99598717854032,
                      [1] 37.567179953757886
                     ]
           ]
         }

Which is indexed correctly.

If you need multi string I guess you need more data and add one more layer of arrays in the ruby script.

user_143672890
Related