I have spark job that get data from a glue table, process it and write it to another glue table.
After I see in the spark ui that all the jobs finished, and in s3 that all the files written successfully, and in the catalog that the new partition exists, the driver keep running for 1 hour and I am not sure what it's doing.
The last logs that I have from the driver are:
Jan 12, 2022 @ 21:10:28.852 Broadcasting large task binary with size 3.4 MiB
Jan 12, 2022 @ 21:10:22.426 Broadcasting large task binary with size 3.3 MiB
Jan 12, 2022 @ 21:10:01.132 Broadcasting large task binary with size 1127.3 KiB
Jan 12, 2022 @ 21:04:31.961 Broadcasting large task binary with size 2000.1 KiB
Jan 12, 2022 @ 20:59:12.680 Broadcasting large task binary with size 1982.6 KiB
Jan 12, 2022 @ 20:59:11.872 Broadcasting large task binary with size 1960.6 KiB
Jan 12, 2022 @ 20:58:49.900 Broadcasting large task binary with size 1959.7 KiB
Jan 12, 2022 @ 20:58:49.131 Broadcasting large task binary with size 1937.7 KiB
Jan 12, 2022 @ 20:58:38.287 Broadcasting large task binary with size 1936.7 KiB
Jan 12, 2022 @ 20:57:24.815 Broadcasting large task binary with size 1195.2 KiB
I check the driver thread dumps and it looks like it stuck on glue-metastore-delegate-0 thread. It changes every couple of seconds from waiting to running.
This is the thread code:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
java.net.SocketInputStream.read(SocketInputStream.java:171)
java.net.SocketInputStream.read(SocketInputStream.java:141)
sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:464)
sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68)
sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1341)
sun.security.ssl.SSLSocketImpl.access$300(SSLSocketImpl.java:73)
sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:957)
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
com.amazonaws.http.protocol.SdkHttpRequestExecutor.doReceiveResponse(SdkHttpRequestExecutor.java:82)
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1236)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
com.amazonaws.services.glue.AWSGlueClient.doInvoke(AWSGlueClient.java:4419)
com.amazonaws.services.glue.AWSGlueClient.invoke(AWSGlueClient.java:4395)
com.amazonaws.services.glue.AWSGlueClient.executeGetPartitions(AWSGlueClient.java:2767)
com.amazonaws.services.glue.AWSGlueClient.getPartitions(AWSGlueClient.java:2743)
com.amazonaws.glue.catalog.metastore.DefaultAWSGlueMetastore.getCatalogPartitions(DefaultAWSGlueMetastore.java:321)
com.amazonaws.glue.catalog.metastore.DefaultAWSGlueMetastore.access$100(DefaultAWSGlueMetastore.java:62)
com.amazonaws.glue.catalog.metastore.DefaultAWSGlueMetastore$2.call(DefaultAWSGlueMetastore.java:280)
com.amazonaws.glue.catalog.metastore.DefaultAWSGlueMetastore$2.call(DefaultAWSGlueMetastore.java:277)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
Any ideas? Thanks