Couchbase - Bulk insert via Java

user_15186569 0
Viewed 208

Introduction

I am currently working on a Java project using Couchbase as a database. As part of it, I came to create a massive data insert via a json file and a csv file.

Question

And I was wondering, what is the best way to achieve this one? Is it possible to have a sample code?

couchbase bulkinsert java
1 Answers

CSV conversion:

Benchmark:

You have several libraries to perform the conversion between CSV <-> Java object, here is a benchmark of the different libraries:

Library Read (rec/sec) Write (rec/sec) Dependencies Size (KiB)
Commons CSV 1,128,102 3,354,703 no 50
FastCSV 4,738,726 5,034,953 no 31
Jackson CSV 3,770,602 3,995,294 yes 2,040
Java CSV 1,922,189 2,732,843 no 13
Opencsv 1,085,935 1,808,982 yes 2,625
Sfm+ASM 5,164,967 1,901,154 yes 1.498
Sfm-ASM 4,652,517 1,901,154 yes 1,498
Super CSV 1,406,090 1,730,984 no 96
Univocity 3,594,900 4,050,255 no 437

For more information on this benchmark you can visit this site: https://github.com/osiegmar/JavaCsvBenchmarkSuite#results

FastCSV:

Iterative reading of some CSV data with a header

NamedCsvReader.builder().build("header 1,header 2\nfield 1,field 2")
    .forEach(row -> row.getField("header 2"));

For more information about FastCSV you can visit this site: https://github.com/osiegmar/FastCSV

JSON conversion:

Benchmark:

You have several libraries to perform the conversion between Json <-> Java object, here is a benchmark of the different libraries:

enter image description here

For more information on this benchmark you can visit this site: https://github.com/ngs-doo/dsl-json

jackson-databind:

Conversion from json string to java object:

String json = "{\"name\":\"Hassan\",\"age\":23}";
Person person = new ObjectMapper().readValue(json, Person.class);

For more information about jackson-databind you can visit this site: https://github.com/FasterXML/jackson-databind

CouchBase insertion:

Example of Bulk Insert in Java:

protected void doWork() {
    final String key = "javaDevguideExampleBulkInsert";
    // Create a JSON document content
    final JsonObject content = JsonObject.create().put("item", "A bulk insert test value");
    // Describe what we want to do asynchronously using RxJava Observables:
    ReactiveCollection reactiveCollection = collection.reactive();
    Flux<MutationResult> resultFlux = Flux.range(0, 10)
            .map(index -> { return key + "_" + index; })
            .flatMap(k -> reactiveCollection.upsert(k, content));
    resultFlux.subscribe(System.out::println);
}

This code is from official docs-sdk-java of CouchBase

user_151865690
Related