Can several esutil.BulkIndexer share an elasticsearch.Client?

user_5508532 0
Viewed 171

I have an array of documents to bulk indexusing the github.com/elastic/go-elasticsearch/v8 library.

Each document may be added to one of a handful of different indexes, so my code is something like:

    var indexers map[string]esutil.BulkIndexer = make(map[string]esutil.BulkIndexer)
    
    for i := range documents {
      var bi esutil.BulkIndexer
      var ok bool
      var err error
      if bi, ok = indexers[documents[i].index]; !ok {
         bi, err = esutil.NewBulkIndexer(esutil.BulkIndexerConfig{
            Index:         documents[i].index,   // The index name
            Client:        client,               // The Elasticsearch client* Same isntance for all indexers*
            NumWorkers:    BULK_NUM_WORKERS,     // The number of worker goroutines
            FlushBytes:    int(BULK_FLUSHBYTES), // The flush threshold in bytes
            FlushInterval: 5 * time.Second,      // The periodic flush interval
        })
        if err != nil {
            fmt.Printf("Error creating the indexer: %s", err)
            continue
        }
        indexers[documents[i].indexName] = bi
    }

The idea is I have a map of esutil.BulkIndexer , an they all share the same elasticsearch.Client instance. So my question is if it's ok and safe for several esutil.BulkIndexer to share the same elasticsearch.Client, or do they each need a separate instance of a elasticsearch.Client ?

elasticsearch go
1 Answers

Could not find the details on official documentation. But Looking at this link it is recommended to share the client across the workers(go routines) for parallel requests

The client connection is created on package initialization, and subsequent requests happening in parallel fashion shares the same client.

from the above link

package httpexample
import (
    "github.com/elastic/go-elasticsearch/v7"
)

var client *elasticsearch.Client

func init() {
    var err error

    ... # Client configuration
    client, err = elasticsearch.NewClient(cfg)
    if err != nil {
        log.Fatalf("elasticsearch.NewClient: %v", err)
    }
}
//This method will be called in parallel
func HttpExample(w http.ResponseWriter, r *http.Request) {
    ... # Client usage
}

Note: I am an expert to give a recommendation on using multiple bulkIndexer.

user_7679490
Related