Loki retention period s3 + dynamo

Viewed 1437

I have a question about the retention mechanism in grafana-loki I need store logs for one year and be able to query them Setup is in k8s with the official Loki chart

below is my config

  auth_enabled: false
  ingester:
    chunk_idle_period: 5m
    chunk_block_size: 262144
    chunk_retain_period: 1m
    # max_transfer_retries: 0
    lifecycler:
      ring:
        kvstore:
          store: inmemory
        replication_factor: 1
  limits_config:
    enforce_metric_name: false
    reject_old_samples: true
    reject_old_samples_max_age: 168h
  schema_config:
    configs:
      - from: 2020-05-15
        store: aws
        object_store: s3
        schema: v11
        index:
          prefix: loki_index_prod_
          period: 8760h
          tags:
            env: prod-loki
  server:
    http_listen_port: 3100
  storage_config:
    aws:
      s3: s3://us-east-1/loki-prod-t
      s3forcepathstyle: true
      dynamodb:
        dynamodb_url: dynamodb://us-east-1
  table_manager:
    index_tables_provisioning:
      inactive_read_throughput: 5
      inactive_write_throughput: 5
      provisioned_write_throughput: 10
      provisioned_read_throughput: 10
    chunk_tables_provisioning:
      inactive_read_throughput: 5
      inactive_write_throughput: 5
      provisioned_write_throughput: 10
      provisioned_read_throughput: 10
    retention_deletes_enabled: true
    retention_period: 8760h
    creation_grace_period: 24h

Now retention is configured for a year, but when the retention period is exited new table in dynamodb will be created, and the old one will be immediately deleted?

In this case, I will lose all logs from the previous year and start from scratch?

Another aspect is how to deal with the retention of chunks in s3?

Please advice ...

1 Answers

No, you'll not lose all logs from the previous year and start from scratch. The Table Manager keeps the last tables alive using the following formula:

number_of_tables_to_keep = floor(retention_period / table_period) + 1

In your case:

number_of_tables_to_keep = 8760 / 8760 + 1 = 2

See the Loki documentation here.

Related