Retrieve selected fields with Spring Data Elasticsearch by using fields option, not by source filtering

khonsu 0
Viewed 28

We're trying to migrate our Opensearch cluster to Elasticsearch.

We've been using Spring Data Elasticsearch with OpenSearch and using fields option to retrieve selected fields. (BaseQueryBuilder#withFields along with BaseQueryBuilder#withSourceFilter)

Sample code:

    NativeSearchQueryBuilder queryBuilder =
        new NativeSearchQueryBuilder().withQuery(query)
                                      .withFields("someId")
                                      .withSourceFilter(new FetchSourceFilter(new String[] {}, new String[] {"*"}));

This was working with Spring Data 4.2.X and Opensearch 1.2.4.

However, with Spring Data 4.4.X and Elasticsearch 8.3, SearchHit's content field does not contain given fields.

What I want to achieve is similar to this query:

GET some_index/_search
{
  "query": {
    "match_all": {}
  },
  "fields": [
    "someId"
  ],
  "_source": false
}

Tried attempts:

1.


    NativeSearchQueryBuilder queryBuilder =
        new NativeSearchQueryBuilder().withQuery(query)
                                      .withFields("someId");

No luck, it's as if this parameter is ignored, returns all the fields in documents.

2.

    NativeSearchQueryBuilder queryBuilder =
        new NativeSearchQueryBuilder().withQuery(query)
                                      .withSourceFilter(new FetchSourceFilter(new String[] {"someId"}, null))

It works. However, in official ES documentation, it states that:

Using fields is typically better

These options are usually not required. Using the fields option is typically the better choice, unless you absolutely need to force loading a stored or docvalue_fields.

So is it worse that using source filtering instead of fields option performance wise?

Is it possible to achieve disabling source and getting selected fields by fields option with Spring Data Elasticsearch?

If it's not possible we consider to use SearchSourceBuilder instead of Spring Data's NativeSearchQueryBuilder.

spring-data-elasticsearch opensearch spring-data elasticsearch
1 Answers

I cannot reproduce that. I just ran a sample application (set up using Spring Boot 2.7.0 with Spring Data Elasticsearch 4.4.2 and an Elasticsearch instance version 8.3.3).

My test entity class is named Foo and the property in this object I use is moreText. The code to send the request:

var query=new NativeSearchQueryBuilder()
    .withQuery(matchAllQuery())
    .withFields("moreText")
    .withSourceFilter(new FetchSourceFilter(new String[]{},new String[]{"*"}))
    .build();

    return operations.search(query, Foo.class);

I trace the calls to Elasticsearch with an intercepting proxy and see that this creates the following call:

{
    "from": 0,
    "size": 10,
    "query": {
        "match_all": {
            "boost": 1.0
        }
    },
    "version": true,
    "explain": false,
    "_source": {
        "includes": [],
        "excludes": [
            "*"
        ]
    },
    "fields": [
        {
            "field": "more-text"
        }
    ]
}

The returned answer is:

{
    "took": 93,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [
            {
                "_index": "foo",
                "_type": "_doc",
                "_id": "42",
                "_version": 1,
                "_score": 1.0,
                "_source": {},
                "fields": {
                    "more-text": [
                        "More text!"
                    ]
                }
            }
        ]
    }
}

The returned answer is exactly what is wanted. This is mapped into the Java entity, see this screenshot from the debugger:

enter image description here

As you can see, Spring Data Elasticsearch also manage if the name of the entitys property (moreText) is different from the field name in Elasticsearch (more-text).

So there must be something different in your setup. Could it be that in your mapping the _source is disabled (or some fields?), see https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html#disable-source-field

Can you provide a minimal sample application on Github that reproduces this behaviour?

P.J.Meisch0
Related