I created a AWS Lambda function that calls FredMeyers store location API. Sending the POST request locally in my IDE works fine but when running within lambda the POST request times out. I can make other GET requests to different Grocery store APIs successfully both locally and in other Lambda functions. Lambda is not running in a VPC.
I'm thinking either one of:
1) my sendPost method is setup incorrectly in that it works only locally
2) API Gateway and Lambda is somehow preventing my POST request so the urlConnection is still waiting (HttpUrlConnection)
3) Target URL destination is somehow is behaving different when the call is made from AWS Lambda
During the lambda run, the code hangs on:
int responseCode = urlConnection.getResponseCode();
HttpHelper:
public class HttpsClient{
private final String urlPath;
public HttpsClient(final String urlPath) {
this.urlPath = urlPath;
}
public String sendPost(final Map<String, String> requestPropertiesMap, final String requestBodyJson) throws IOException {
URL obj = new URL(urlPath);
HttpURLConnection urlConnection = (HttpURLConnection) obj.openConnection();
urlConnection.setRequestMethod("POST");
if (true) {
SSLSocketFactory socketFactory = (SSLSocketFactory) SSLSocketFactory.getDefault();
((HttpsURLConnection) urlConnection).setSSLSocketFactory(socketFactory);
}
requestPropertiesMap.entrySet().stream().forEach(entry ->
urlConnection.addRequestProperty(entry.getKey(), entry.getValue()));
final byte[] bodyRequest = requestBodyJson.getBytes(StandardCharsets.UTF_8);
urlConnection.setDoOutput(true);
DataOutputStream outputStream = new DataOutputStream(urlConnection.getOutputStream());
outputStream.write(bodyRequest);
outputStream.flush();
outputStream.close();
int responseCode = urlConnection.getResponseCode();
if (responseCode == HttpURLConnection.HTTP_OK) { // success
try {
final String encoding = urlConnection.getContentEncoding();
final InputStream in;
if ("gzip".equals(encoding)) {
in = new GZIPInputStream(urlConnection.getInputStream());
} else {
in = urlConnection.getInputStream();
}
return textFromInputStream(in);
} finally {
urlConnection.getInputStream().close();
}
} else {
throw new RuntimeException("Something went wrong... :( status code: " +
responseCode + " " + urlConnection.getResponseMessage());
}
}
private String textFromInputStream(final InputStream inputStream) {
return new BufferedReader(new InputStreamReader(inputStream, StandardCharsets.UTF_8)).lines()
.collect(Collectors.joining("\n"));
}
}
Fred Meyer Store Search
public class FredMeyerStoreSearch implements StoreSearchClient {
private final String apiURLFormat = "https://www.fredmeyer.com/atlas/v1/modality/options";
@Override
public List<Store> searchStores(final String zipCode) {
final String requestBodyJson = new Gson().toJson(StoreOptionsRequest.builder()
.address(StoreAddress.builder().postalCode(zipCode).build())
.build());
try {
final String fullUrl = apiURLFormat;
final HttpsClient httpsClient = new HttpsClient(fullUrl);
final String response = httpsClient.sendPost(getRequestParameters(), requestBodyJson);
return new FredMeyerStoreTransformer(response).createStores();
} catch (IOException e) {
throw new RuntimeException("Something wrong with search call", e);
}
}
private Map<String, String> getRequestParameters() {
return new HashMap<String, String>() {{
put("cookie", Constant.EMPTY_COOKIE);
put("accept", "application/json, text/plain, */*");
put("accept-encoding", "gzip, deflate, br");
put("accept-language", "en-US,en;q=0.9");
put("sec-fetch-dest", "empty");
put("sec-fetch-mode", "cors");
put("sec-fetch-site", "same-origin");
put("user-agent", Constant.USER_AGENT);
put("content-type", "application/json");
}};
}
@Builder
@Data
private static class StoreOptionsRequest {
private StoreAddress address;
}
@Builder
@Data
private static class StoreAddress {
private String postalCode;
}
}
Edit: 05/23 @ 3:30AM PST
- I refactored to use OkHttp3 HttpClient instead of HttpUrlConnection and the same behavior
- I introduced a network interceptor (OkHttp3) to log request/responses
- Both local and lambda GET/POST requests are identical.
- The local POST request hits the nginx load balancer and works fine
- The lambda POST request hits the AkamaiGHost caching server which seems to be configured to deny POST requests
Anyone have insight on this behavior and potential workarounds?
Local:
Received response for https://www.fredmeyer.com/atlas/v1/modality/options in 329.7ms
Server: nginx
Content-Type: application/json; charset=utf-8
Vary: Accept-Encoding
Access-Control-Allow-Origin: https://www.fredmeyer.com
Surrogate-Control: no-store
Vary: Origin
X-Content-Type-Options: nosniff
X-Dns-Prefetch-Control: off
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Vcap-Request-Id: c19c969d-0b1d-47c7-5784-c035a800c9c6
X-Xss-Protection: 1; mode=block
Content-Encoding: gzip
Content-Length: 3457
Expires: Sat, 23 May 2020 10:02:41 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Sat, 23 May 2020 10:02:41 GMT
Connection: keep-alive
Server-Timing: cdn-cache; desc=MISS
Server-Timing: edge; dur=58
Server-Timing: origin; dur=143
X-KT-ACR-LMA-CT: true
Strict-Transport-Security: max-age=31536000 ; includeSubDomains
Set-Cookie:<hiding>
Set-Cookie:<hiding>
Lambda:
Received response for https://www.fredmeyer.com/atlas/v1/modality/options in 201.3ms
Server: AkamaiGHost
Mime-Version: 1.0
Content-Type: text/html
Content-Length: 304
Expires: Sat, 23 May 2020 Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Sat, 23 May 2020 Connection: close
Server-Timing: cdn-cache; desc=HIT
Server-Timing: edge; dur=1
X-KT-ACR-LMA-CT: true
Strict-Transport-Security: max-age=31536000 ; includeSubDomains
Decoded Response: <HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>
You don't have permission to access "http://www.fredmeyer.com/atlas/v1/modality/options" on this server.<P>
Reference #18.1e032417.1590228331.1335abc8
</BODY>
</HTML>