How to query Apache-Pulsar?

user_10077504 0
Viewed 727

Apache pulsar offers a very interesting architecture with the tiered storage offloaders.

I wonder how I could do interactive queries from another application? I mean direct queries to the key-value system/"database" and not using pulsar-sql, which uses Presto underneath.

In @sijieg, on twitter, has posted the following schema : enter image description here

It looks like I can access the State-store (or even the Segment-reader) and directly access the data in the Bookies (and maybe the tiered storage, according a Metastore ?) How can we access these State-store/Segment-reader and access the data as would do Flink-Pulsar or Spark-Pulsar.

apache-pulsar
2 Answers

A fast answer is "You don't directly query Apache Pulsar". But let's have a deeper look.

Apache Pulsar is not an RDBMS where SQL queries are the main way how to work with data. If your system needs SQL queries and the load is not so extreme, just use a traditional RDBMS or NoSQL of your choice.

Why is it hard to make queries to Apache Pulsar? The main reason is that Apache Pulsar is a distributed pub-sub messaging system, where data are treated as unbounded streams and it makes it hard to run traditional SQL queries in a performant way. The solution, in this case, is steam processing engines (Pulsar Functions, Apache Flink, Apache Spark) where data can be selected, transformed and written somewhere.

If you still need to run queries against some data stored in Pulsar, it is possible to forward this information to RDBMS or NoSQL database using built-in sink connectors.

And for analytics, it can be enough to use pulsar-sql communicating directly with the storage layer (Bookies).

user_16065770

I am not sure why you are opposed to using pulsar-sql, which uses Presto. That is the preferred method for performing complex SQL based queries against data that is stored inside the BookKeeper storage layers (both on bookie disk and in tiered storage). Presto parses the SQL and produces the AST and query plan to return the data, so it does provide a lot of value in that regard.

However, if you are interested in accessing the data directly on BK, then you can use the older DLog API. or the new table (key/value) service that is embedded in Bookies.

user_32293010
Related