Active Record transforming dates inconsistently when querying a postgresql view

Viewed 132

I've got a rails app that is creating a view of calendar events, some of which are stored as calendar events with a starts_at column and some of which have a generated starts_at column created from a repeating schedule.

The view is a union and looks like this (simplified):

(
  SELECT
    'appointment' AS event_type,
    starts_at
  FROM
    appointments
)
UNION
(
  SELECT
    'schedule' AS event_type,
    (
      to_timestamp(
        CONCAT(
          start_date,
          ' ',
          lpad(start_hour :: text, 2, '0'),
          ':',
          lpad(start_minute :: text, 2, '0'),
          ':00.000000'
        ),
        'YYYY-MM-DD hh24:mi:ss:us'
      ) at time zone 'UTC'
    ) :: timestamp without time zone AS starts_at
  FROM
    schedule_items
)

This works fine and when I query the view in postgres I get:

 event_type  |          ends_at
-------------+----------------------------
 schedule    | 2021-10-18 08:00:00
 schedule    | 2021-11-08 09:00:00
 appointment | 2021-10-14 17:44:15.122543

These are all correct times in UTC not in the local timezone.

I wrapped an ActiveRecord model around this view (using the scenic gem to generate the view) but when I query the model, it provides a correct time for the appointment record but an incorrect time for the schedule (generated) records.

The appointment is shown at the UTC time above (current 1 hour behind the local UK timezone).

The schedule time is show at the time above plus 1 hour (in UTC) so is 2 hours ahead of UTC when cast by Active Record.

If I build a custom cast_type for the attribute I can see that it's reading the first time above as 2021-10-18 08:00:00 UTC (effectively converting it to local time but tagging as UTC) but for the appointment record it is correctly reading it as 2021-10-14 17:44:15.122566 UTC.

If I use a basic SQL query in Active Record I get the following result:

irb(main):045:0> r = ActiveRecord::Base.connection.execute('select starts_at from calendar_events')
irb(main):045:0> r[0]
=> {"starts_at"=>2021-10-18 09:00:00.000000 UTC}
irb(main):046:0> r[2]
=> {"starts_at"=>2021-10-14 17:44:15.122543 UTC}

Which is showing that the time is being parsed wrongly for the first record and correctly for the last one.

If I use the pg gem natively I get the same result as if I query using sql:

irb(main):001:0> conn = PG.connect( dbname: 'my_db' )
=> #<PG::Connection:0x00000001222d3a48>
irb(main):002:1* conn.exec('select * from calendar_events') do |result|
irb(main):003:2*   result.each do |row|
irb(main):004:2*     puts row['starts_at']
irb(main):005:1*   end
irb(main):006:0> end
2021-10-14 17:44:15.122543
2021-11-08 09:00:00
2021-10-18 08:00:00

which shows the results I'm expecting.

The datetime column types in the calendar table are the same type as used to create the datetime in the view, e.g.

starts_at timestamp without time zone

Both the rails app and the Postgres db are set to the Europe/London timezone and all timestamps are written to the db as timestamp without time zone types with the value in UTC.

I have tried any number of ways to resolve it (e.g. creating a custom cast_type for the attribute on the model, adding self.skip_time_zone_conversion_for_attributes, attribute_before_type_cast['starts_at']) none of which solve the problem that ActiveRecord appears to be converting some dates from UTC to a local time but still marking them as UTC.

So am at a bit of a loss so any suggestions anyone has would be gratefully received!

1 Answers

Ok so in case anyone else runs into this issue here's the solution:

  1. Ensure the timezone in postgresql.conf is set to 'Etc/UTC' (not your local timezone which the installer may default to - e.g. installing with homebrew on OSX defaults to your local TZ)
  2. When generating the dates in the view use this form of casting:
SELECT
  (
    to_timestamp(
      '2021-11-18 09:00:00', 'YYYY-MM-DD HH24::m1:ss'
    ) at time zone 'UTC' at time zone 'Europe/London'
  ) :: timestamp without time zone as starts_at;

Then ActiveRecord sorts everything out as you would expect. Obviously the timezone setting on postgres does seem to interact with ActiveRecord in this case.

Doing raw queries in psql makes it look like the data is the same but it's clearly not exactly the same when read from ActiveRecord using the detail session timezone of UTC.

Timezones are indeed hard. If my local machine had been set to 'Etc/UTC' as all my production systems are then there would have been no issue.

So lesson is if you do install postgres locally for rails development and want to use generated dates in views it's important to ensure that the timezone is set to 'Etc/UTC'.

That was indeed tough to work out but thanks are due to @eyeslandic who suggested a bug report on rails which then led me to create a test script to demonstrate the issue.

Once I then had the right combination of settings it was easy to validate it using that script. Would have been hard to get to the bottom of it otherwise.

Related