populate the session id and generate new session id if timestamp difference two consecutive event is more then 30 min

Viewed 148

Input - read from existing hive or redshift table

user   |    Timestamp    |  SessionId
---------------------------------------
u1     |    10:00AM      |      ?    
u1     |    10:05AM      |      ?    
u1     |    10:10AM      |      ?    
u1     |    10:15AM      |      ?    
u1     |    11:40AM      |      ?    
u1     |    11:50AM      |      ?    
u1     |    12:15PM      |      ?

Expected output

user   |    Timestamp    |  SessionId
---------------------------------------
u1     |    10:00AM      |      s1    
u1     |    10:05AM      |      s1    
u1     |    10:10AM      |      s1    
u1     |    10:15AM      |      s1    
u1     |    11:40AM      |      s2    
u1     |    11:50AM      |      s2    
u1     |    12:15PM      |      s3

How are we going to resolve this to update existing table using hive or redshift ?

1 Answers

Convert timestamp to unix_timestamp (seconds), get previous timestamp using lag() function, calculate difference and assign new_session=1 if more than 30 min passed, calculate running sum of new_session to get session id.

with mydata as (
select 'u1' as `user`,'10:00AM' `timestamp` union all    
select 'u1','10:05AM' union all    
select 'u1','10:10AM' union all    
select 'u1','10:15AM' union all    
select 'u1','11:40AM' union all    
select 'u1','11:50AM' union all    
select 'u1','12:15AM' -----------15 min after midnight
)

select `user`, `timestamp`, 
       concat('s',sum(new_session) over(partition by `user` order by `timestamp`)) as session_id
from 
(
select --calculate new_session flag based on differennce between ts and prev_ts
      `user`, `timestamp`, ts, prev_ts,
      case when ((ts-prev_ts)/60 > 30) or prev_ts is NULL then 1 end as new_session
from      
(
select `user`, `timestamp`, ts, 
      --calculate previous time
       lag(ts) over(partition by `user` order by ts) prev_ts
from
(
--convert time to seconds
select `user`, `timestamp`, unix_timestamp(`timestamp`,'hh:mma') as ts from mydata
)s --ts conversion
)s --prev_ts
)s --new_session

Result:

user    timestamp   session_id
u1      10:00AM      s1
u1      10:05AM      s1
u1      10:10AM      s1
u1      10:15AM      s1
u1      11:40AM      s2
u1      11:50AM      s2
u1      12:15AM      s3

Note that I changed 12:15PM to 12:15AM to get s3 session because in your data example 12:15PM is a 15 minutes after noon, the difference between 11:50AM and 12:15PM is 25 minutes and does not trigger new session. To get S3 session like in your question it should be 12:15AM. 12:15AM is a 15 minutes after midnight, see 12-hour_clock wiki

Related