Recently my elb environment have decided to not respond to some of the requests. It happens periodically and gets fixed somehow when restart the app servers. I am getting below warning in my "var/log/healthd/daemon.log"-
W, [2018-11-03T05:29:56.816559 #7415] WARN -- : discarding statistic item after validation error (Invalid timestamp): {:id=>"0", :namespace=>"application", :timestamp=>1541222570, :data=>"{\"duration\":10,\"latency_histogram\":[[59.201,1],[59.522,1],[59.556,1],[59.696,1]],\"http_counters\":{\"status_499\":4,\"request_count\":4}}"}
I did some searching on this issue and found that it is related to vpc but i am not using one.
Nginx logs have reported HTTP 499 errors. but i guess that is because client is rejecting the requests because of timeout.
Nodejs logs just have some unresponded requests and no errors related to the code or anything else.
Note - I don't know if this is relevant or not but the requests failing are mostly the requests which have some kind of database query.