Best way to join different continuos views in TimescaleDB

Viewed 164

This is a Postgres database with Timescale and PostGIS installed.

There are 3 Hypertables with different information about status of ships. position information, voyage information, vessel details information. Example:

CREATE TABLE position(
    imo         int                    NOT NULL,
    "timeReceived" TIMESTAMPTZ            NOT NULL,
    location    geography              NOT NULL,
    speed       DOUBLE PRECISION  NULL
);

SELECT create_hypertable('position', 'timeReceived');

CREATE TABLE voyage(
    imo         int                    NOT NULL,
    "timeReceived"        TIMESTAMPTZ            NOT NULL,
    destination text   NOT NULL
);

SELECT create_hypertable('voyage', 'timeReceived');


CREATE TABLE details (
    imo         int                    NOT NULL,
    "timeReceived" TIMESTAMPTZ            NOT NULL,
    draught     double precision       NULL
);

SELECT create_hypertable('details ', 'timeReceived');

totally these 3 tables have around 40 columns. Updated with 10 million rows a day on my most updated table and 1 million on the 2 least updated. Currently it's just beggining so there are around 150 million rows on the biggest table.

On each of these hypertables there is a continuous aggregate which is update hourly. example:

CREATE materialized VIEW position_hourly WITH (timescaledb.continuous)
AS      SELECT
        time_bucket('1 hour', "timeReceived"),
        imo,
        last("timeReceived", "timeReceived") "timeReceived",
        last(src, "timeReceived") src,
        last("position","timeReceived")::geography "position"
    from position s_position
    group by imo, time_bucket('1 hour', "timeReceived")
    WITH NO DATA;

SELECT add_continuous_aggregate_policy('position_hourly',
  start_offset => INTERVAL '1 year',
  end_offset => INTERVAL '1 hour',
  schedule_interval => INTERVAL '3 hours');

In the end I want to create a view/table with all the information from all these sources in 1 table. What I tried:

create materialized view all_data as
SELECT *
FROM position_hourly ph 
LEFT JOIN LATERAL (
    SELECT ---just example columns
         destination, 
         vh."timeReceived" voyage_timereceived
    FROM voyage_hourly vh
    WHERE ph.imo = vh.imo 
    AND ph."time_bucket" >= vh."time_bucket"
    ORDER BY  vh."time_bucket" DESC
    LIMIT 1
    ) vh ON TRUE
LEFT JOIN LATERAL (
    SELECT ---just example columns
          draught, 
          vvh."timeReceived" vessel_timereceived
    FROM vessel_hourly vvh
    WHERE ph.imo = vvh.imo 
    AND ph."time_bucket" >= vvh."time_bucket"
    ORDER BY  vvh."time_bucket" DESC
    LIMIT 1
    ) vvh ON TRUE

(note that the query is much larger due to the extra columns but the logic is the same)

I am able to write a query and preview it with a limit 100 clause on it, but when It tries to materialize the view it takes 5+ hours to complete. This view should be updated hourly.

How can I optimize my query so it loads faster ie. within 10 minutes? Should I consider some changes in my setup?

Thank you

1 Answers

A few ideas:

  1. Have you enabled compression? Probably you can also segment by imo column and get some good compression levels and also it will make it easier to get results looking for only chunks with the same segment.
  2. Can you make the hypertable a bit lean and move some of these extra columns to a secondary table? Maybe you'll not query all the data all the time and it will make it easier to fetch only what you need.
  3. Can you move the LEFT JOIN LATERAL to a CTE?
  4. Have you tried to run a mix of batch inserts and then refresh_continuous_aggregate? Maybe you can open a secondary connection to keep refreshing in parallel?
Related