How can you tail AWS application load balancer logs? Is it really this hard? If so, why?

user_64911 0
Viewed 54

I'm new to AWS's Application Load Balancers, and I'm feeling a bit lost coming from the nginx world. Is it true that there's no way to tail your request logs? If so, are there relatively simple ways people use to watch their logs?

I read the docs that you can send the logs to S3, and from there, I guess you can set up Athena our ElasticSearch or something, but boy, that sounds like an all-day or multiple-day project! I don't want to have to download logs to my laptop and unzip them either, just to monitor things.

I just want to tail some logs in realtime. Is it really this hard?

And if it really is this hard, I'd love to know why. Most times when AWS surprises me, there's some reason related to scaling or something, but I can't imagine why tailing some logs is so hard.

Thank you!

aws-application-load-balancer logging
1 Answers

It really is that hard. API Gateway, by comparison, can write its access logs to CloudWatch Logs, so you get closer to real-time.

The reason is that files on S3 are written atomically, so you have a choice of either batching records together into a single file, or writing one file per record. The latter would (1) add an extreme amount of overhead to a high-volume production deployment, and (2) be almost impossible to analyze due to the large number of reads that would be needed. With the original ELB, iirc you could only write logs on 60 minute intervals; the 5 minute intervals were added later.

Plus, the only time that it makes sense to actually tail logs is in development.

If you have an enterprise-level support contract, it would be worth telling your TAM that this is important to you. If enough people do so, then sooner or later they'll implement it (probably by writing logs to CloudWatch). Or maybe they've already implemented it, and your TAM can get you into the feature preview.

user_134007290
Related