data in ksqlDB table are persistent?

user_12409368 0
Viewed 282

We are using ksqlDB 0.11.0, and we want to persistently store data in ksqlDB's table. But we set log.retention.hours to 72, so i confused whether the table store the recent 72 hours data?

create stream stream_ori_sacmes_pack_detail with (kafka_topic = 'sacmes.pack.pack_detail',value_format = 'avro'); 

create stream STREAM_ETL_PACK_DETAIL
with (kafka_topic='gp_etl_pack_detail',PARTITIONS=1, REPLICAS=3, VALUE_FORMAT='avro')
as 
select 
id,
line_id,
tenant_id
from STREAM_ORI_SACMES_PACK_DETAIL
partition by id
emit changes;

create table KSQL_TABLE_GP_PACK_DETAIL
(id varchar,
line_id varchar,
tenant_id INTEGER
)
WITH (KAFKA_TOPIC='gp_etl_pack_detail', VALUE_FORMAT='avro');
ksqldb
1 Answers

By default, the latest data in a table will always be retained (beyond the 72 hours in your case). So, for each primary key in the table, the latest key+value pair (row) will always be available.

Imagine the following changes are being applied to the table (via a stream of changes):

| Key   | City  | Age      | ... |
==================================
| Alice | Paris | 24 years | ... |  INSERT
| Alice | Turin | 24 years | ... |  UPDATE
| Bob   | Bern  | 40 years | ... |  INSERT
| Alice | Rome  | 24 years | ... |  UPDATE

At the end of this change stream, here's what the table will look like:

| Key   | City  | Age      | ... |
==================================
| Alice | Rome  | 24 years | ... |
| Bob   | Bern  | 40 years | ... |

This latest data for the table will always be retained. But previous changes ("older" rows for a given key), like Alice having been in Paris and Turin before, will eventually be purged from the system because of the limited retention.

user_17435800
Related