How to use SearchAfter API correctly?

Viewed 4284

I recently work with elasticsearch and I have such a question. I have a million documents in index and I wanna to get more than 10_000. For this I can use scroll API or SearchAfter API. I understood how does scroll api works but I have some problem with SearchAfter.

Here is my SearchSourceBuilder method:

public SearchRequest buildRequest(SearchDistanceParameters args) {
    final SearchSourceBuilder searchSourceBuilder = prepareSearchSourceBuilder(args);
    final SearchRequest searchRequest = new SearchRequest();
    return searchRequest.source(searchSourceBuilder);
}

private SearchSourceBuilder prepareSearchSourceBuilder(SearchDistanceParameters searchDistanceParameters) {
    final FieldSortBuilder fieldSortBuilder = new FieldSortBuilder("_id").order(SortOrder.ASC);
    final SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
    final GeoDistanceQueryBuilder geoDistanceQueryBuilder = geoDistanceQuery(GeoLocationModelFieldName.LOCATION.name().toLowerCase());
    geoDistanceQueryBuilder.point(searchDistanceParameters.getLatitude(), searchDistanceParameters.getLongitude());
    geoDistanceQueryBuilder.distance(searchDistanceParameters.getDistance(), DistanceUnit.KILOMETERS);
    searchSourceBuilder.query(geoDistanceQueryBuilder);
    searchSourceBuilder.sort(fieldSortBuilder);
    searchSourceBuilder.searchAfter();
    return searchSourceBuilder;
}

Here I do sort before searchAfter() as mention in SearchAfter API doc.

Here I am sending my request to ElasticSearch:

public SearchResponse sendRequestToElastic(SearchDistanceParameters args) throws IOException {
    SearchRequest searchRequest = searchByDistanceRequestBuilder.buildRequest(args);
    return elasticDao.search(searchRequest, RequestOptions.DEFAULT); // standard RestHighLevelClient.search method inside elasticDao.
}

And finally I am trying to get my objects from SearchResponse:

public List<GeoPointsFromElasticSearchResponse> searchByDistance(SearchDistanceParameters searchDistanceParameters) {
        final SearchResponse searchResponse = searchRepository.searchByDistance(searchDistanceParameters);
        return getGeoPointsFromElasticSearchResponses(searchResponse);
    }

private List<GeoPointsFromElasticSearchResponse> getGeoPointsFromElasticSearchResponses(SearchResponse searchResponse) {
        SearchHit[] hits = searchResponse.getHits().getHits();
        return Arrays.stream(hits)
                .map(hit -> {
                    final GeoPointsFromElasticSearchResponse geoPointsFromElasticSearchResponse = new GeoPointsFromElasticSearchResponse();
                    final Map<String, Object> sourceMap = hit.getSourceAsMap();
                    final Map map = (Map) sourceMap.get(GeoLocationModelFieldName.LOCATION.name().toLowerCase());
                    geoPointsFromElasticSearchResponse.setLatitude((Double) map.get("lat"));
                    geoPointsFromElasticSearchResponse.setLongitude((Double) map.get("lon"));
                    log.info("Sorted hits: {}", hit.getSortValues());
                    return geoPointsFromElasticSearchResponse;
                }).collect(Collectors.toList());
    }

But I've got only 10_000 objects. It seems I am doing something wrong in last part. What am I doing wrong? How to use SearchAfter api in java correctly?

1 Answers
Related