How to apply filter only if nested mapping exists

Question

I'm trying to apply location radius on nested ES query but the nested value is not present all the time causing exception

"[nested] nested object under path [contact.address] is not of nested type"

I tried to check if the property exists then apply filter but nothing worked so far

The mapping is like:

{
  "records": {
    "mappings": {
      "user": {
        "properties": {
          "user_id": {
            "type": "long"
          },
          "contact": {
            "properties": {
              "address": {
                "properties": {
                  "city": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "location": {
                    "properties": {
                      "lat": {
                        "type": "long"
                      },
                      "lng": {
                        "type": "long"
                      },
                      "lon": {
                        "type": "text",
                        "fields": {
                          "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                          }
                        }
                      }
                    }
                  }
              },
              "email": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "name": {
                "properties": {
                  "first_name": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "last_name": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  }
                }
              }
            }
          },
          "created_at": {
            "type": "date"
          }
        }
      }
    }
  }
}

and sometimes the records do not have the location or address data which cases problems. sample record:

{
   "contact": {
      "name": {
        "first_name": "Test",
        "last_name": "User"
       },
        "email": "test@user.com",
        "address": {}
    },
    "user_id": 532188
}

here is what i'm trying:

GET records/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "exists": {
            "field": "contact.address"
          }
        },
        {
          "exists": {
            "field": "contact.address.location"
          }
        }
      ],
      "minimum_should_match": 1,
      "should": [
        {
          "bool": {
            "filter": {
              "nested": {
                "ignore_unmapped": true,
                "path": "contact.address",
                "query": {
                  "geo_distance": {
                    "distance": "50mi",
                    "contact.address.location": {
                      "lat": 51.5073509,
                      "lon": -0.1277583
                    }
                  }
                }
              }
            }
          }
        }
      ]
    }
  }
}

user_4039431 · Answer

You need to define proper mapping with nested datatype to avoid this issue, looks dynamic mapping is creating some issue.

I defined my own mapping with nested datatype and even when I miss, some data in the nested fields, it doesn't complain.

Index def

{
    "mappings": {
        "properties": {
            "user_id": {
                "type": "long"
            },
            "contact": {
                "type": "nested"
            }
        }
    }
}

Index sample doc

{
   "contact": {
      "name": {
        "first_name": "raza",
        "last_name": "ahmed"
       },
        "email": "opster@user.com",
        "address" :{ --> note empty nested field
        }

    },
    "user_id": 123456
}

Index another doc with data in the nested field

{
   "contact": {
      "name": {
        "first_name": "foo",
        "last_name": "bar"
       },
        "email": "opster@user.com",
        "address": {
            "location" :{. --> note nested data as well
                "lat" : 51.5073509,
                "lon" : -0.1277583
            }
        }
    },
    "user_id": 123456
}

Index another doc, which doesn't have even empty `nested` data

{
   "contact": {
      "name": {
        "first_name": "foo",
        "last_name": "bar"
       },
        "email": "opster@user.com"
    },
    "user_id": 123456
}

Search query using `nested` field

{
    "query": {
        "nested": {
            "path": "contact", --> note this
            "query": {
                "bool": {
                    "must": [
                        {
                            "exists": {
                                "field": "contact.address"
                            }
                        },
                        {
                            "exists": {
                                "field": "contact.name.first_name"
                            }
                        }
                    ]
                }
            }
        }
    }
}

The search result doesn't complain about the docs which don't include the nested doc (query which gives you issues)

"hits": [
      {
        "_index": "nested",
        "_type": "_doc",
        "_id": "3",
        "_score": 2.0,
        "_source": {
          "contact": {
            "name": {
              "first_name": "foo",
              "last_name": "bar"
            },
            "email": "opster@user.com",
            "address": { --> note the nested doc
              "location": {
                "lat": 51.5073509,
                "lon": -0.1277583
              }
            }
          },
          "user_id": 123456
        }
      }