I have a java (AWS Java SDK 1.11.313) SQS consumer polling a queue every 20 seconds which has been running for nearly a week. Over this time it has failed on two occasions, both times throwing the following exception:
com.amazonaws.SdkClientException: Unable to execute HTTP request: Permission denied: connect
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1114)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1064)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at com.amazonaws.services.sqs.AmazonSQSClient.doInvoke(AmazonSQSClient.java:2033)
at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:2009)
at com.amazonaws.services.sqs.AmazonSQSClient.executeReceiveMessage(AmazonSQSClient.java:1485)
at com.amazonaws.services.sqs.AmazonSQSClient.receiveMessage(AmazonSQSClient.java:1460)
at net.myCompany.messageHandlers.QMessageHandler.receiveMsg(QMessageHandler.java:112)
at net.myCompany.MessageHandler.receiveDirectMsg(MessageHandler.java:23)
at net.myCompany.commandProcessors.QCommandProcessor.run(QCommandProcessor.java:23)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketException: Permission denied: connect
at java.net.DualStackPlainSocketImpl.connect0(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:83)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339)
at com.amazonaws.http.conn.ssl.SdkTLSSocketFactory.connectSocket(SdkTLSSocketFactory.java:132)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.amazonaws.http.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:76)
at com.amazonaws.http.conn.$Proxy9.connect(Unknown Source)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1236)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
... 14 more
On both occasions the client recovered after ~15mins. The method below is responsible for creating and sending the receive request. The exception arose on the sqs.receiveMessage() method call above the for loop. Each time a request is sent a new instance of com.amazonaws.services.sqs.AmazonSQS is created and assigned to the sqs variable.
List<JSONObject> receiveMsg() {
List<JSONObject> cmdMsgList = new ArrayList<>();
try {
if(sqs != null) {
System.out.println("Attempting message receive at: " + Instant.now());
// Receive messages
ReceiveMessageRequest receiveMessageRequest = new ReceiveMessageRequest(commandQUrl);
receiveMessageRequest.setWaitTimeSeconds(20);
receiveMessageRequest.setMaxNumberOfMessages(10);
List<Message> messages = sqs.receiveMessage(receiveMessageRequest).getMessages();
for (Message message : messages) {
JSONObject msgBody = checkTargetDevice(message);
if (msgBody != null) {
cmdMsgList.add(msgBody);
}
}
} else {
ServiceManager.getInstance().setServerConnection(false);
ServiceManager.getInstance().setConnectionStatus("Can't receive message - Message channel does not exist");
}
} catch (QueueDoesNotExistException mia) {
ServiceManager.getInstance().setServerConnection(false);
}catch (AmazonServiceException ase) {
if(ServiceManager.getInstance().isDebug()) {
ase.printStackTrace();
ase.getErrorMessage();
}
} catch (AmazonClientException ace) {
if(ServiceManager.getInstance().isDebug()) {
ace.printStackTrace();
}
}
return cmdMsgList;
}
I thought maybe too many connections were being created and not closed properly however after using netstat it doesn't seem excessive with only a few tcp connections for that program. I have since added the sqs.shutdown() call before the return statement however I'm not expecting much change since the AWS documentation states that this method call on the AmazonSQS interface is an optional method, and callers are not expected to call it.
Any insights or advice on this problem is greatly appreciated. Thanks in advance.