Results changed after sorting a table

Viewed 44

I have a following scenario:

I created a batch job using SQL API.

final TableEnvironment tEnv = TableEnvironment.create(EnvironmentSettings.newInstance().inBatchMode().build());

I load the data from csv files, convert/aggregate it using SQL API.

At some stage I have a table:

CREATE VIEW ohlc_current_day as
SELECT
    CAST(transact_time as DATE) as `day`,
    instrument_id,
    first_value(price) as `open`,
    min(price) AS `low`,
    max(price) AS `high`,
    last_value(price) as `close`,
    count(*) AS `count`,
    sum(quantity) AS volume,
    sum(quantity * price) AS turnover
FROM trades //table loaded from csv
group by CAST(transact_time as DATE), instrument_id

Now when check the results:

select * from ohlc_current_day where instrument_id=14
+------------+---------------+---------+---------+--------+---------+-------+-----------+---------------+
|        day | instrument_id |    open |     low |   high |   close | count |    volume |      turnover |
+------------+---------------+---------+---------+--------+---------+-------+-----------+---------------+
| 2021-04-11 |            14 |  1723.0 |  1709.0 | 1743.0 |  1728.0 |   679 |  487470.0 |   8.4114803E8 |
+------------+---------------+---------+---------+--------+---------+-------+-----------+---------------+

The results are repeatable and correct (checked with reference).

Then, for futrher processing, I need ohlc values from the previous day which are already stored in a database:

CREATE TABLE ohlc_database (
  `day` TIMESTAMP,
  instrument_id INT,
  `open` float,
  `low` FLOAT,
  `high` FLOAT,
  `close` FLOAT,
  `count` BIGINT,
  volume FLOAT,
  turnover FLOAT
) WITH (
 'connector' = 'jdbc',
 'url' = 'url',
 'table-name' = 'ohlc',
 'username' = 'user',
 'password' = 'password'
)

Let's now merge ohlc_current_day with ohlc_database:

CREATE VIEW ohlc_raw as
SELECT * from ohlc_current_day
UNION ALL
select
    CAST(`day` as DATE) as `day`,
    instrument_id,
    `open`,
    `low`,
    `high`,
    `close`,
    `count`,
    volume,
    turnover
FROM ohlc_database
WHERE `day` = '2021-04-10' //hardcoded previous day date

And check the results:

select * from ohlc_raw where instrument_id=14
+------------+---------------+--------+--------+--------+---------+-------+-----------+---------------+
|        day | instrument_id |   open |    low |   high |   close | count |    volume |      turnover |
+------------+---------------+--------+--------+--------+---------+-------+-----------+---------------+
| 2021-04-10 |            14 | 1696.0 | 1654.0 | 1703.0 |  1691.0 |   936 | 1040888.0 |  1.74619264E9 |
| 2021-04-11 |            14 | 1723.0 | 1709.0 | 1743.0 |  1728.0 |   679 |  487470.0 |   8.4114829E8 |
+------------+---------------+--------+--------+--------+---------+-------+-----------+---------------+

results are ok, values the same as in previous select query.

Now let's order by day:

CREATE VIEW ohlc as
SELECT * from ohlc_raw ORDER BY `day`

Check the results:

select * from ohlc where instrument_id=14
+------------+---------------+-----------------+-----------------+-------------+----------------+----------------------+--------------------------------+--------------------------------+
|        day | instrument_id |            open |             low |        high |          close |                count |                         volume |                       turnover |
+------------+---------------+-----------------+-----------------+-------------+----------------+----------------------+--------------------------------+--------------------------------+
| 2021-04-10 |            14 |          1696.0 |          1654.0 |      1703.0 |         1691.0 |                  936 |                      1040888.0 |                   1.74619264E9 |
| 2021-04-11 |            14 |          1729.0 |          1709.0 |      1743.0 |         1732.0 |                  679 |                       487470.0 |                    8.4114854E8 |
+------------+---------------+-----------------+-----------------+-------------+----------------+----------------------+--------------------------------+--------------------------------+

open and close are wrong compared to previous values. They are calculated using first_value() and last_value() functions which depend on the order of elements. So my guess is that order by in last query has changed the order and this is why there are different results.

Is my understanding correct? How can I fix it?

1 Answers

I thought that first_value() or last_value() themself is a sort operation. When you order by in last query, the sort according to last_value() is out of order. May be you can output the result into database after union all, and then do order by date after extract data from database if possible.

Related