Apache HttpClient 4.5.13 java.net.SocketException: Broken pipe (Write failed)

Question

I have a Java 8 Http client application that communicates with a server that will reject requests under a few circumstances (e.g. invalid or missing Authorization header, payload larger than 1MB). When a request is rejected I see expected HTTP response codes when using cURL (e.g. 401 Unauthorized, 403 Forbidden, 413 Payload Too Large). However, when using the Apache HttpClient I am getting a java.net.SocketException: Broken pipe (Write failed). I believe this happens when the client wants to write the request body but the server has already closed the connection without fully reading the request body.

Ideally, I'd like to get the server's response so I can handle the "real" error (e.g. inform the user their request is too large, or let them know their session is no longer valid). Without the response from the server I would have to assume that this is a network error and retry. Is there any way to gracefully handle the java.net.SocketException: Broken pipe (Write failed) exception and still read the server response?

Example cURL request

$> printf 'x%.0s' {1..1048577} | curl -i --data @- https://my.server.com/anything
HTTP/1.1 100 Continue

HTTP/1.1 413 Request Entity Too Large
Content-Type: application/json
X-Frame-Options: DENY
Content-Length: 92
Connection: Close

Example Code

import org.apache.http.HttpResponse;
import org.apache.http.client.entity.EntityBuilder;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;

public class Main {

    public final static void main(String[] args) throws Exception {
        CloseableHttpClient client = HttpClients.createDefault();
        try {
            HttpPost request = new HttpPost("https://my.server.com/anything");
            String largePayload = new String(new char[1024*1024+1]).replace('\0', 'x'); // >1MB
            request.setEntity(EntityBuilder.create().setText(largePayload).build());
            HttpResponse response = client.execute(request);
            System.out.println(response.getStatusLine());
        } finally {
            client.close();
        }
    }

}

Full Stack Trace

Exception in thread "main" java.net.SocketException: Broken pipe (Write failed)
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
    at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
    at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
    at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:886)
    at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:857)
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
    at org.apache.http.impl.conn.LoggingOutputStream.write(LoggingOutputStream.java:74)
    at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124)
    at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:160)
    at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113)
    at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:120)
    at org.apache.http.entity.StringEntity.writeTo(StringEntity.java:167)
    at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
    at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152)
    at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
    at Main.main(main.java:15)

user_15263819 · Answer

I was facing a similar issue, I fixed it including the following configuration to the httpClient

RequestConfig requestConfig = RequestConfig.custom()
            .setExpectContinueEnabled(true).build();
HttpClientBuilder clientBuilder = HttpClientBuilder.create()
            .setDefaultRequestConfig(requestConfig);
CloseableHttpClient httpClient = clientBuilder.build();

And in the next link you can found the explanation to do it http://httpcomponents.10934.n7.nabble.com/Broken-pipe-Write-failed-when-making-Unauthorized-request-td34235.html.

Edit: The link above is broken so here is a similar link that works: https://lists.apache.org/thread/p38b3lrjt4vhyqthwhqcq3vsx3dpr3wj

Here is the summary that Joe asked for: The scenario is that a server rejects a request based on headers and closes the connection so the client gets a broken pipe error. The expect-continued header tells the client to look for a 100 status indicating that the server has accepted the headers and the client should send the rest of the data. Note that not all servers appear to do this correctly and can still close the connection after sending the 100 status.

user_2771322 · Answer

Explanation about ExpectContinueEnabled is as follows

This abstract class serves as a foundation for all HTTP methods that support 'Expect: 100-continue' handshake.

The purpose of the 100 (Continue) status (refer to section 10.1.1 of the RFC 2616 for more details) is to allow a client that is sending a request message with a request body to determine if the origin server is willing to accept the request (based on the request headers) before the client sends the request body. In some cases, it might either be inappropriate or highly inefficient for the client to send the body if the server will reject the message without looking at the body.

'Expect: 100-continue' handshake should be used with caution, as it may cause problems with HTTP servers and proxies that do not support HTTP/1.1 protocol.

Reference: https://hc.apache.org/httpclient-legacy/apidocs/org/apache/commons/httpclient/methods/ExpectContinueMethod.html

Apache HttpClient 4.5.13 java.net.SocketException: Broken pipe (Write failed)

2 Answers

Explanation about ExpectContinueEnabled is as follows