When a message is formatted as json, it is automatically turned into attributes. It seems like attributes cant be queried without first being turned into facets (which only applies to new log lines, and means you sometimes have to see something show up, then facetize it, then debug it).
Is there a way to query the message directly, bypassing the attribute facet requirement?
Actually it is possible, but you need to put every JSON log into single quotes or add some prefix before each log, so that Datadog agent will consider this as a 'text'. E.g. log.json file should contain quoted or prefixed logs to be treated as plain text:
'{"key1": "value1"}'
'{"key1": "value2"}'
'{"key1": "value3"}'
!{"key1": "value1"}
!{"key1": "value2"}
!{"key1": "value3"}
After that, in Datadog Logs Configuration, you need to add a pipeline with Grok parser filter json (see filter tab in Matcher and Filter):
This allowed me to perform full text search thru all fields in my JSON logs and automatically parse all JSON fields as attributes.
P.S. This solution was provided by Datadog support 2 years ago. And seems they are working on solution to allow full text search for JSON logs.