could you please support me on stupid question:
I have some Java class:
public class ProbePoint implements Serializable, Cloneable {
private long arrivalTimeMillis = 0;
private long captureTimeMillis = 0;
//...
}
public class Trip implements Serializable, Cloneable {
private ArrayList<ProbePoint> points = new ArrayList<>();
//...
}
I have Dataset[Trip]. I need to collect some min/max values. What would be better implementation of next:
public class DataRanges implements Serializable {
private long minCaptureTs;
private long maxArrivalTs;
}
val timesDs: Dataset[DataRanges] = trips.mapPartitions(t => {
var minCaptTime = Long.MaxValue
var maxArrTime = Long.MinValue
t.foreach(f => {
if (f.points.head < minCaptTime) minCaptTime = f.points.head
if (f.points.last.getArrivalTimeMillis > maxArrTime) maxArrTime = f.points.last.getArrivalTimeMillis
})
Iterator[DataRanges](
new DataRanges(minStartTime, maxEndTime, minArrTime, maxArrTime))
})(Encoders.bean(classOf[DataRanges]))
val times = timesDs.agg(min("minCaptureTs"), max("maxArrivalTs")).head()
}
}