I'm using the Google Maps Geocode API (https://github.com/googlemaps/google-maps-services-java) in a Dataflow job. My DoFn prepares the GeoApiContext at Setup. The process element function is done like so:
public void processElement(ProcessContext c) {
String address = c.element().get("Address").toString();
String id = c.element().get("Id").toString();
Gson gson = new GsonBuilder().create();
try {
GeocodingResult[] results = GeocodingApi.newRequest(this.geocodeContext).address(address).language("pt-BR").components(ComponentFilter.country("BR")).await();
if(results.length == 0) {
TableRow outputRow = new TableRow();
outputRow.set("Id", id);
c.output(outputRow);
} else {
for(GeocodingResult r : results) {
TableRow outputRow = convertTableRow(gson.toJson(r).toString());
outputRow.set("Id", id);
c.output(outputRow);
}
}
} catch(ApiException e) {
LOGGER.error("ApiException on address: {}", address, e);
} catch(InterruptedException e) {
LOGGER.error("InterruptedException on address: {}", address, e);
} catch(IOException e) {
LOGGER.error("IOException on address: {}", address, e);
}
}
This code worked fine locally, but when deployed to dataflow it throws a network error:
exception: "java.net.ConnectException: Failed to connect to maps.googleapis.com/2607:f8b0:4001:c05:0:0:0:5f:443
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:265)
at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:183)
at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.java:224)
at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.java:108)
at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.java:88)
at okhttp3.internal.connection.Transmitter.newExchange(Transmitter.java:169)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:41)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229)
at okhttp3.RealCall$AsyncCall.execute(RealCall.java:172)
at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.ConnectException: Network is unreachable (connect failed)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:591)
at okhttp3.internal.platform.Platform.connectSocket(Platform.java:130)
at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:263)
... 22 more
I've ensured that the VM spawned has internet access and I can even ping the maps.googleapis.com endpoint from inside the container:
USER@test-geocode-07020834-qmrj-harness-3k2l ~ $ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b2fd123138aa 3a1cb7aedd54 "/opt/google/dataflo…" 6 minutes ago Up 5 minutes k8s_healthchecker_dataflow-test-geocode-07020834-qmrj-harness-3k2l_default_5648e9815f2ca5beea8b0eb945e12d1f_0
086e36c3dd23 4127911f4769 "/opt/google/dataflo…" 6 minutes ago Up 5 minutes k8s_vmmonitor_dataflow-test-geocode-07020834-qmrj-harness-3k2l_default_5648e9815f2ca5beea8b0eb945e12d1f_0
2890fa415af5 664bd8972b23 "/opt/google/dataflo…" 6 minutes ago Up 6 minutes k8s_shuffle_dataflow-test-geocode-07020834-qmrj-harness-3k2l_default_5648e9815f2ca5beea8b0eb945e12d1f_0
eea757bf6be7 gcr.io/cloud-dataflow/v1beta3/beam-java11-batch "/opt/google/dataflo…" 6 minutes ago Up 6 minutes k8s_java-batch_dataflow-test-geocode-07020834-qmrj-harness-3k2l_default_5648e9815f2ca5beea8b0eb945e12d1f_0
b636784118f5 k8s.gcr.io/pause:3.1 "/pause" 6 minutes ago Up 6 minutes k8s_POD_dataflow-test-geocode-07020834-qmrj-harness-3k2l_default_5648e9815f2ca5beea8b0eb945e12d1f_0
lucas@test-geocode-07020834-qmrj-harness-3k2l ~ $ docker exec -it eea /bin/sh
# ping maps.googleapis.com
PING maps.googleapis.com (172.217.214.95) 56(84) bytes of data.
64 bytes from 172.217.214.95: icmp_seq=1 ttl=115 time=1.08 ms
64 bytes from 172.217.214.95: icmp_seq=2 ttl=115 time=1.28 ms
64 bytes from 172.217.214.95: icmp_seq=3 ttl=115 time=1.15 ms
64 bytes from 172.217.214.95: icmp_seq=4 ttl=115 time=1.41 ms
^C
--- maps.googleapis.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 1.089/1.235/1.414/0.131 ms
#
Regarding versions, I'm using the latest beam version (2.22.0) and the latest google maps version (0.14.0).
No idea what else to look at, and any help is appreciated.
UPDATE
The problem seems to be the fact that the request is done with an ipv6 address. However, GCE machines seems to have no support for ipv6 and the call simply fails without falling back to ipv4.
Considering that, there doesn't seem to be any way out of this problem:
- Configuring the JVM to prefer ipv4 address can't be done with Dataflow (JVM flags are ignored)
- There's also no way to customize the GCE machine (since a base Dataflow image is used)
- The library doesn't seem to open any options to configure ipv4 or ipv6
Thanks