How can I improve performance processing stream that is filtered based on other stream?

Viewed 145

I'm facing a performance problem while processing a big stream of objects (from one source) that is being filtered and mapped with values from another big stream/collection (suppose from a different source). I'm trying to do a kind of Join (sql).

My machine is taking 11+ minutes to execute it.

I have tried added a filter before to the map, but it degraded more the situation.

What could I do on this matter to get better results?

I'm going to provide an example of what I'm trying to achieve.

Note that the filter is using just the ID, but could be used more common properties of both streams.

import static java.util.stream.Collectors.toSet;
import java.io.IOException;
import java.time.Duration;
import java.time.Instant;
import java.util.Collection;
import java.util.Objects;
import java.util.Random;
import java.util.stream.IntStream;
import java.util.stream.Stream;

public class ProcessorQuestion {

  static record Element(int id, String content) {

    public Element(int id, String content) {
      this.id = id;
      this.content = content == null ? "Data " + id : content;
    }
  }

  static record Row(int id, String content) {

    public Row(int id, String content) {
      this.id = id;
      this.content = content == null ? "Row " + id : content;
    }
  }

  static record RowVsElement(Row row, Element element) {
  }

  private static Random r = new Random();

  protected static Stream<Element> loadElementsData() {
    return IntStream.range(1, 1_000_000)
                    .mapToObj(value -> new Element(r.nextInt(235_000), null));
  }

  protected static Stream<Row> loadRowsData() {
    return IntStream.range(1, 235_000)
                    .mapToObj(value -> new Row(r.nextInt(235_000), null));
  }

  public static void main(String[] args) throws IOException {

    var init = Instant.now();

    final ProcessorQuestion processor = new ProcessorQuestion(loadElementsData());
    processor.process();

    System.err.println("Runned in " + Duration.between(init, Instant.now()).toMinutes() + " min");

  }

  private final Collection<Element> entries;

  public ProcessorQuestion(Stream<Element> entries) {
    this.entries = entries.collect(toSet());
  }

  void process() {
    //    System.out.println("rows size = " + rows.size());
    System.out.println("elements size = " + entries.size());

    loadRowsData().parallel()
                  //                    .filter(r0 -> entries.stream()
                  //                                                   .anyMatch(
                  //                                                       entry -> entry.getId() == r0.getId()))
                  .map(r1 -> entries.parallelStream()
                                    .filter(entry -> entry.id() == r1.id())
                                    .findFirst()
                                    .map(elem -> new RowVsElement(r1, elem))
                                    .orElse(null))
                  .filter(Objects::nonNull)
                  .forEachOrdered(pair -> saveOnMedia(pair.row, pair.element));

  }

  void saveOnMedia(Row row, Element element) {

    StringBuilder rsb = new StringBuilder(Integer.toString(row.id()));

    rsb.append(Integer.toString(row.id()));
    rsb.append(";");
    rsb.append(Integer.toString(element.id()));
    rsb.append(";");
    rsb.append(row.content());
    rsb.append(";");
    rsb.append(element.content());
    rsb.append(System.lineSeparator());

    System.out.println(rsb.toString());

  }
}

I've taken some print from the screen of the execution on VisualVM:

Thread CPU Time CPU Samples Samples Memory Heap

1 Answers

Try using the map, as people suggested:

Map<Integer, Element> elementMap = entries.stream().collect(Collectors.toMap(it -> it.id(), it -> it, (a, b) -> a));

loadRowsData()
        //                    .filter(r0 -> entries.stream()
        //                                                   .anyMatch(
        //                                                       entry -> entry.getId() == r0.getId()))
        .map(r1 -> elementMap.containsKey(r1.id()) ? new RowVsElement(r1, elementMap.get(r1.id())) : null)
        .filter(Objects::nonNull)
        .forEachOrdered(pair -> saveOnMedia(pair.row, pair.element));

It should be completed in one second if you just save the result to a list, instead of printing all the result. Printing out the result may take couple of seconds.

As Knittl said in the comments: "Your algorithm has O(m*n) complexity – with m=235000 and n=1000000, that's ~235000000000". That's the reason it takes so long.

Related