Flink read Hive by Streaming Mode, why only the latest partition be read, even if i set the start offset as the oldest partition?

Viewed 125

this is the partitions formart of the hive table:

  year=2021/month=09/day=15
  year=2021/month=09/day=16
  year=2021/month=09/day=17
  year=2021/month=09/day=18
  year=2021/month=09/day=19
  year=2021/month=09/day=20

this is the sql :

SELECT * FROM table_name
OPTIONS(
  'streaming-source.enable'='true',
  'streaming-source.monitor-interval'='1 min',
  'streaming-source.partition.include'='all',
  'stream-source.consume-order'='partition-name',
  'streaming-source.consume-start-offset'='year=2021/month=09/day=15'
)

the result:

I set start offset as year='year=2021/month=09/day=15', but only the data of partition='year=2021/month=09/day=20' will flow in Flink, why???

0 Answers
Related