What could cause socket ConnectException: Connection timed out?

user_13792 0
Viewed 58444

We have a Webstart client that communicates to the server by sending serialized objects over HTTPS using java.net.HttpsURLConnection.

Everything works perfectly fine on my local machine and on test servers located in our office, but I'm experiencing a very, very strange issue which is only occurring on our production and staging servers (and sporadically at that). The main difference I know of between those servers and the ones in our office is that they are located elsewhere and client-server communication with them is considerably slower, but it worked fine for a long time in production prior to this as well.

Anyway, here's what's happening:

  • The client, after setting options such as read timeout and properties such as Content-Type on the HttpURLConnection, calls getOutputStream() on it to get the stream to write to.
  • At this point, from what I can tell, the client hangs for some period of time.
  • The client then throws the following exception:
java.net.ConnectException: Connection timed out: connect
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(Unknown Source)
    at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
    at java.net.PlainSocketImpl.connect(Unknown Source)
    at java.net.SocksSocketImpl.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(Unknown Source)
    at com.sun.net.ssl.internal.ssl.BaseSSLSocketImpl.connect(Unknown Source)
    at sun.net.NetworkClient.doConnect(Unknown Source)
    at sun.net.www.http.HttpClient.openServer(Unknown Source)
    at sun.net.www.http.HttpClient.openServer(Unknown Source)
    at sun.net.www.protocol.https.HttpsClient.(Unknown Source)
    at sun.net.www.protocol.https.HttpsClient.New(Unknown Source)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(Unknown Source)

Note that this is not a SocketTimeoutException, which the connect() method on HttpURLConnection says it throws if the timeout expires before a connection can be established. Also, when this happens I am able to call conn.getResponseCode() and I get a response code of 200.

  • On the server side, an EOFException is thrown in ObjectInputStream's constructor, which tries to read the serialization header but fails because the client never gets the OutputStream to write to.

In case it helps, here are the calls being made on the HttpsURLConnection prior to the call to getOutputStream() (edited to show only the calls being made rather than the whole structure of the code doing this):

HttpsURLConnection conn = (HttpsURLConnection) url.openConnection();
conn.setUseCaches(false);
conn.setReadTimeout(30000);
conn.setRequestProperty("Cookie", cookie);
conn.setDoOutput(true);
conn.setRequestProperty("Content-Type", "application/x-java-serialized-object");
conn.getOutputStream();

The thing is, I have no idea how any of this could be happening, especially given that it only happens occasionally (no clear pattern of activity that I can tell) and even then only when there's (relatively) high latency between the client and the server.

Given what I've been able to find so far about java.net.ConnectException: Connect timed out, I wondered if it weren't some network or firewall issue on the network our servers are running on... but that doesn't make much sense to me given that the request is clearly getting through to the servlet. Also, other apps running on the same network have not reported similar issues.

Does anyone have any idea what the cause of this could be, or even what I should investigate?

httpurlconnection https sockets exception java
2 Answers

We also experience sporadic timeouts when using it on our servers. We are able to fix it with two things:

  1. Use specific ContentLength via setFixedLengthStreamingMode (brought down the error rate from ~150 to 10)
  2. Retry if a timeout occurs (Error rate from 10 to 0. After max. one retry everything went through)

pseudo code: 

//set timeouts to 6s
try{
 //open connection here and write etc.
 //use a timeout of 6s (since retry is in place)
} 
catch (java.io.InterruptedIOException e) {
 //read- or connection time out try again                 
}

Another theory why this is happening could be the following:

In the documentation of the HttpURLConnection/HttpsURLConnection one can read the following:

Each HttpURLConnection instance is used to make a single request but the underlying network connection to the HTTP server may be transparently shared by other instances.

So now calling close() only would be ok but also calling disconnect() would terminate the socket for the other users / transparently shared connections which would then run into a SocketTimeOut after the timeout period is reached.

user_23115280
Related