This is a Postgres database with Timescale and PostGIS installed.
There are 3 Hypertables with different information about status of ships. position information, voyage information, vessel details information. Example:
CREATE TABLE position(
imo int NOT NULL,
"timeReceived" TIMESTAMPTZ NOT NULL,
location geography NOT NULL,
speed DOUBLE PRECISION NULL
);
SELECT create_hypertable('position', 'timeReceived');
CREATE TABLE voyage(
imo int NOT NULL,
"timeReceived" TIMESTAMPTZ NOT NULL,
destination text NOT NULL
);
SELECT create_hypertable('voyage', 'timeReceived');
CREATE TABLE details (
imo int NOT NULL,
"timeReceived" TIMESTAMPTZ NOT NULL,
draught double precision NULL
);
SELECT create_hypertable('details ', 'timeReceived');
totally these 3 tables have around 40 columns. Updated with 10 million rows a day on my most updated table and 1 million on the 2 least updated. Currently it's just beggining so there are around 150 million rows on the biggest table.
On each of these hypertables there is a continuous aggregate which is update hourly. example:
CREATE materialized VIEW position_hourly WITH (timescaledb.continuous)
AS SELECT
time_bucket('1 hour', "timeReceived"),
imo,
last("timeReceived", "timeReceived") "timeReceived",
last(src, "timeReceived") src,
last("position","timeReceived")::geography "position"
from position s_position
group by imo, time_bucket('1 hour', "timeReceived")
WITH NO DATA;
SELECT add_continuous_aggregate_policy('position_hourly',
start_offset => INTERVAL '1 year',
end_offset => INTERVAL '1 hour',
schedule_interval => INTERVAL '3 hours');
In the end I want to create a view/table with all the information from all these sources in 1 table. What I tried:
create materialized view all_data as
SELECT *
FROM position_hourly ph
LEFT JOIN LATERAL (
SELECT ---just example columns
destination,
vh."timeReceived" voyage_timereceived
FROM voyage_hourly vh
WHERE ph.imo = vh.imo
AND ph."time_bucket" >= vh."time_bucket"
ORDER BY vh."time_bucket" DESC
LIMIT 1
) vh ON TRUE
LEFT JOIN LATERAL (
SELECT ---just example columns
draught,
vvh."timeReceived" vessel_timereceived
FROM vessel_hourly vvh
WHERE ph.imo = vvh.imo
AND ph."time_bucket" >= vvh."time_bucket"
ORDER BY vvh."time_bucket" DESC
LIMIT 1
) vvh ON TRUE
(note that the query is much larger due to the extra columns but the logic is the same)
I am able to write a query and preview it with a limit 100 clause on it, but when It tries to materialize the view it takes 5+ hours to complete. This view should be updated hourly.
How can I optimize my query so it loads faster ie. within 10 minutes? Should I consider some changes in my setup?
Thank you