How can i use groupBitmapAnd with AggregatingMergeTree engine in clickhouse?

user_16437150 0
Viewed 667

I'd like to maintain a table in clickhouse which use bitmap "AND" logic to merge bitmaps accross the rows with the same tag_id.

Since bitmap is defined as AggregateFunction(groupBitmap, UInt*) in clickhouse and groupBitmapAnd takes bitmap as its argument, i created a table like:

CREATE TABLE test.bitmap_column_expr_test2
(
    `tag_id` String,
    `z` AggregateFunction(groupBitmapAnd, AggregateFunction(groupBitmap, UInt32))
)
ENGINE = AggregatingMergeTree()
ORDER BY tag_id
SETTINGS index_granularity = 8192

In my case, i want to insert data like:

INSERT INTO test.bitmap_column_expr_test2 VALUES ('tag1', groupBitmapAndState(bitmapBuild(cast([1,2,3,4] as Array(UInt32)))));
INSERT INTO test.bitmap_column_expr_test2 VALUES ('tag1', groupBitmapAndState(bitmapBuild(cast([1] as Array(UInt32)))));
INSERT INTO test.bitmap_column_expr_test2 VALUES ('tag1', groupBitmapAndState(bitmapBuild(cast([1,3,4] as Array(UInt32)))));

And i expect to get the bitmap AND result by:

SELECT bitmapToArray(groupBitmapAndMergeState(z)) FROM test.bitmap_column_expr_test2;

However, my ddl is rewrited by clickhouse as:

CREATE TABLE test.bitmap_column_expr_test2
(
    `tag_id` String,
    `z` AggregateFunction(groupBitmap, AggregateFunction(groupBitmap, UInt32))
)
ENGINE = AggregatingMergeTree()
ORDER BY tag_id

which losts its origin definition of column z

Besides, the insertion will end up with exception:

DB::Exception: Aggregate function groupBitmapAndState(bitmapBuild(CAST([1, 2, 3, 4], 'Array(UInt32)'))) is found in wrong place in query: While processing groupBitmapAndState(bitmapBuild(CAST([1, 2, 3, 4], 'Array(UInt32)'))) (version 20.11.4.13 (official build))

I am not sure if i am doing the right thing to get bitmap accross the rows merged by "AND" logic in AggregatingMergeTree.

clickhouse
1 Answers

groupBitmapAnd accepts as a parameter the bitmap represented as AggregateFunction(groupBitmap, UInt_) type.

ClickHouse doesn't support nested aggregation types (in this case it be AggregateFunction(groupBitmapAnd, AggregateFunction(groupBitmap, UInt_))).

So it needs to follow the pattern described in the official docs - https://clickhouse.tech/docs/en/sql-reference/aggregate-functions/reference/groupbitmapand/#groupbitmapand:

CREATE TABLE bitmap_column_expr_test2
(
    tag_id String,
    z AggregateFunction(groupBitmap, UInt32)
)
ENGINE = MergeTree
ORDER BY tag_id;

INSERT INTO bitmap_column_expr_test2 VALUES ('tag1', bitmapBuild(cast([1,2,3,4,5,6,7,8,9,10] as Array(UInt32))));
INSERT INTO bitmap_column_expr_test2 VALUES ('tag2', bitmapBuild(cast([6,7,8,9,10,11,12,13,14,15] as Array(UInt32))));
INSERT INTO bitmap_column_expr_test2 VALUES ('tag3', bitmapBuild(cast([2,4,6,8,10,12] as Array(UInt32))));

SELECT groupBitmapAnd(z) FROM bitmap_column_expr_test2 WHERE like(tag_id, 'tag%');
┌─groupBitmapAnd(z)─┐
│               3   │
└───────────────────┘
user_3032980
Related