I'm reading a file using TextIO.read() how can I get the line number of the file Or how to maintain our own distributed counter so that I can access the last value and set into the pojo.
Eg: File looks like
TRANSACTION_ID|CLIENT_ID|PRODUCT_CODE
123|1001|2001
124|1002|2002
After reading this content I have one transform which parse into Java pojo class and if you see the pojo one field is lineNumber and then same pojo I'm inserting into database.
class BatchData {
private Integer transactionId;
private Integer clientId;
private Integer productCode;
private Integer lineNumber;
}
Do we have any way to get the line number directly.
What I tried but didn't work.
- First I tried to create my own custom counter using locking and while parsing the file into POJO then I'm calling this getAndInc() method.
public class CustomizedCounter implements Serializable {
private static final long serialVersionUID = 368687587598L;
private static int counter = 0;
private final ReentrantLock lock = new ReentrantLock();
public int getAndInc() {
lock.lock();
try {
counter++;
} finally {
lock.unlock();
}
return counter;
}
}
This didn't work with big file when there are multiple workers.
- Then I though to use the Metrics counter which is provided by the apache beam but we can only use it for maintaining the counter not the actual lineNumber of the file. But my requirement will be fine even if I successfully able to maintain the distributed counter. I took the ref. of below doc which provides 3 types of counter but I can't able to access the last value of the counter at the runtime.
[https://beam.apache.org/documentation/programming-guide/#metrics][1]