I'm facing a performance problem while processing a big stream of objects (from one source) that is being filtered and mapped with values from another big stream/collection (suppose from a different source). I'm trying to do a kind of Join (sql).
My machine is taking 11+ minutes to execute it.
I have tried added a filter before to the map, but it degraded more the situation.
What could I do on this matter to get better results?
I'm going to provide an example of what I'm trying to achieve.
Note that the filter is using just the ID, but could be used more common properties of both streams.
import static java.util.stream.Collectors.toSet;
import java.io.IOException;
import java.time.Duration;
import java.time.Instant;
import java.util.Collection;
import java.util.Objects;
import java.util.Random;
import java.util.stream.IntStream;
import java.util.stream.Stream;
public class ProcessorQuestion {
static record Element(int id, String content) {
public Element(int id, String content) {
this.id = id;
this.content = content == null ? "Data " + id : content;
}
}
static record Row(int id, String content) {
public Row(int id, String content) {
this.id = id;
this.content = content == null ? "Row " + id : content;
}
}
static record RowVsElement(Row row, Element element) {
}
private static Random r = new Random();
protected static Stream<Element> loadElementsData() {
return IntStream.range(1, 1_000_000)
.mapToObj(value -> new Element(r.nextInt(235_000), null));
}
protected static Stream<Row> loadRowsData() {
return IntStream.range(1, 235_000)
.mapToObj(value -> new Row(r.nextInt(235_000), null));
}
public static void main(String[] args) throws IOException {
var init = Instant.now();
final ProcessorQuestion processor = new ProcessorQuestion(loadElementsData());
processor.process();
System.err.println("Runned in " + Duration.between(init, Instant.now()).toMinutes() + " min");
}
private final Collection<Element> entries;
public ProcessorQuestion(Stream<Element> entries) {
this.entries = entries.collect(toSet());
}
void process() {
// System.out.println("rows size = " + rows.size());
System.out.println("elements size = " + entries.size());
loadRowsData().parallel()
// .filter(r0 -> entries.stream()
// .anyMatch(
// entry -> entry.getId() == r0.getId()))
.map(r1 -> entries.parallelStream()
.filter(entry -> entry.id() == r1.id())
.findFirst()
.map(elem -> new RowVsElement(r1, elem))
.orElse(null))
.filter(Objects::nonNull)
.forEachOrdered(pair -> saveOnMedia(pair.row, pair.element));
}
void saveOnMedia(Row row, Element element) {
StringBuilder rsb = new StringBuilder(Integer.toString(row.id()));
rsb.append(Integer.toString(row.id()));
rsb.append(";");
rsb.append(Integer.toString(element.id()));
rsb.append(";");
rsb.append(row.content());
rsb.append(";");
rsb.append(element.content());
rsb.append(System.lineSeparator());
System.out.println(rsb.toString());
}
}
I've taken some print from the screen of the execution on VisualVM:



