NLog/Elasticsearch writes to DataStream not index

Viewed 125

I have NLog configuration that pushes data to Elastic. When the index pattern is static everything works fine. When I change it to be time-based, I can see messages coming to datastream (with the name of the index from configuration), but this datastream is creating .ds-* index which does not follow the name from the pattern. Therefore logs are not visible in Kibana. I have action.auto_create_index even set to true. What am I missing to have this working properly?

Here is NLog.config part related to elastic:

<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      autoReload="true">

    <!-- enable asp.net core layout renderers -->
    <extensions>
        <add assembly="NLog.Web.AspNetCore"/>
        <add assembly="NLog.Targets.ElasticSearch"/>
        <add assembly="NLog.StructuredLogging.Json"/>
    </extensions>

    <variable name="AspData" value="${when:when=length('${aspnet-request-url}') > 0:inner=|url\: ${aspnet-request-url}}${when:when=length('${aspnet-mvc-action}') > 0:inner=|action\: ${aspnet-mvc-action}}" />
    <variable name="ExceptionLayout" value="${onexception:${newline}EXCEPTION - ${exception:format=type,message,method:maxInnerExceptionLevel=5:innerFormat=shortType,message,method}}" />
    <variable name="LoggingLayout" value="${longdate}|EV[${event-properties:item=EventId_Id}]|TH[${threadname:whenEmpty=${threadid}}]|${mdc:userId}|${uppercase:${level}}|${logger}|${message} ${AspData} ${ExceptionLayout}" />
    <!-- the targets to write to -->
    <targets>
        <!--<default-wrapper xsi:type="BufferingWrapper" bufferSize="100"/>--> 
        <!-- write logs to file  -->
        <target name="asyncElastic" xsi:type="AsyncWrapper">
            <target name="elasticLog" xsi:type="ElasticSearch" index="logs-${configsetting:item=ElasticLog.ServiceName}-${date:format=yyyy-MM-dd}"
                uri="${configsetting:item=ElasticLog.ServerUrl}" requireAuth="true" username="${configsetting:item=ElasticLog.User}" password="${configsetting:item=ElasticLog.Password}"
                layout ="${LoggingLayout}" >
                <field name="data" layout="${structuredlogging.json}" layoutType="System.Object" />
                <field name="serviceName" layout="${configsetting:item=ElasticLog.ServiceName}" layoutType="System.String" />
                <field name="userId" layout="${mdc:userId}" layoutType="System.String" />
            </target>
        </target>
    </targets>

    <!-- rules to map from logger name to target -->
    <rules>
        <!--Skip non-critical Microsoft logs and so log only own logs-->
        <logger name="Microsoft.EntityFrameworkCore.Database.Command" maxlevel="Debug" final="true" />
        <logger name="Microsoft.EntityFrameworkCore.Database.Command" maxlevel="Warning" writeTo="asyncElastic" final="true" />
        <logger name="Elastic.Apm" maxlevel="Info" final="true" />
        <logger name="*" minlevel="Trace" writeTo="asyncElastic" />
    </rules>
</nlog>

Edit: After further investigation I found out that default Kibana installation has index template called logs with pattern logs-*-*. This template treats this indice with that pattern as datastream and not index. So if I wanted to have indexes all I had to do is change my index name to not follow that pattern.

0 Answers
Related