AWS CloudFront - forward User-Agent but don't cache against it

You can use Lambda@Edge function (https://docs.aws.amazon.com/lambda/latest/dg/lambda-edge.html) assigned to your CloudFront distribution. You would need two functions:

1. Viewer-Request event handler, that will read User-Agent header and copy it to e.g. X-My-User-Agent. Viewer-Request handler is invoked before the request from the client reaches your Cloudfront Distribution.
2. Origin-Request event handler, that will read X-My-User-Agent and replace User-Agent. Origin-Request handler is invoked when Cloudfront did not find requested page in its cache and sends the request to the origin.

Please note that you should NOT add User-Agent to Cloudfront whitelist:

You can configure CloudFront to cache objects based on values in the Date and User-Agent headers, but we don't recommend it. These headers have a lot of possible values, and caching based on their values would cause CloudFront to forward significantly more requests to your origin.

Ref: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html#request-custom-headers-behavior

Example of Viewer-Request handler (Lambda@Edge can be written only in NodeJS or Python, Ref: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-requirements-limits.html#lambda-requirements-lambda-function-configuration):

'use strict';

exports.handler = (event, context, callback) => {
  const request = event.Records[0].cf.request;
  const headers = request.headers;
  const customUserAgentHeaderName = 'X-My-User-Agent';
  const userAgent = headers['user-agent'][0].value;

  headers[customUserAgentHeaderName.toLowerCase()] = [
    {
      key: customUserAgentHeaderName,
      value: userAgent
    }
  ];


  callback(null, request);
};

Example of Origin-Request handler:

'use strict';

exports.handler = (event, context, callback) => {
  const request = event.Records[0].cf.request;
  const headers = request.headers;
  const customUserAgentHeaderName = 'X-My-User-Agent';
  const realUserAgent = headers[customUserAgentHeaderName.toLowerCase()][0].value;

  headers['user-agent'] = [
    {
      key: 'User-Agent',
      value: realUserAgent
    }
  ];


  callback(null, request);
};

It seems that around July 2020 Amazon introduced new feature "Origin Request policies". You should be able to use it (without bogging down in Lambda@Edge): https://aws.amazon.com/blogs/networking-and-content-delivery/amazon-cloudfront-announces-cache-and-origin-request-policies/

TLDR:

Over time, we’ve seen numerous cases in which the new functionality could be useful for customers. Examples such as:

• Forwarding information such as the User-Agent to the origin for analytics/logging but without serving different content variants based on device type (now you can forward the user-agent header and exclude it from the cache-key)

^ this is your use-case :)

If the requests are cached across different user-agents, in case of a hit, the real-user agent will not be passed to the origin at all. CloudFront will just return the cached response.

You mentioned that you like to send the user-agent information to Elasticsearch. Unless you are only interested in the requests that are missed, you can not rely on the logs collected from the origin application.

If you have Lambda@Edge to send user-agent as realUserAgent, but the user-agent header is itself not a caching parameter, the origin will still not receive that data in case of a Miss.

The only solution that I see here, is to use the access logs generated from CloudFront. The CloudFront access logs contain not only user-agent but also IP addresses and other useful information. This data is logged for both Hit and Miss. It is also easy to set up a logstash to send this information to Elasticsearch.

• [1] https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html
• [2] https://aws.amazon.com/premiumsupport/knowledge-center/cloudfront-logs-elasticsearch/

This might be a simple solution. If you wanted User-Agent for a unique type of URLs example, /tracking/a,/tracking/b like this create a new distribution for this path [tracking*] and whitelist User-Agent for this distribution only. So you are not messing with AWS caching for all urls but for this path only.