How change date Format column in csv.bz2 and update file in Unix

user_3617165 0
Viewed 90

I have a bz2 compressed csv file with millions of records.The first row contains the file header. I have multiple date columns but there are only two specific columns that i need to change to Format the date from yyyy/mm/dd hh24:mm and set it to dd/mm/yy hh24:mm and update the csv.bz2 file. I was looking researching with awk, with this

bzcat OBACountryIPTV_CALICUX_Report_Session_Local_Timezone_2021-07-02_000000.csv.bz2 |
head -1 | sed 's /; / \ n / g' | nl

I managed to find the position of the columns that I need to reformat:

  1  playtime
  2  traffic
  .
  .
 30  end_time
 31  start_time
 .
 .
122  playback_id

The output of each row for example:

0.0;0.0;;0.0;0;;0.0;0;0.0;0.0;7.0;-1.0;0.0;0.0;0;0;0;0;0;0;0;0;0.0;0.0;0.0;;;0.0;0.0;2021/05/11 17:52;2021/05/11 17:52;0.0;Argentina;;;;LATAM;Unknown;WIFI;Undefined;;Undefined;Streaming;TELEFE;N/A - Multicast;;LIVE;;STB;;;;;;;;Not reported;;;29_2255508521_1620777131852;{"channel_id": 3346, "channel_name": "TELEFE", "channel_number": null, "delaytime": null, "dist_id": null, "player_session_name": null, "playtime": null, "program_id": 500508915, "resolution": "HD", "device_version": "2020.06.26.00.08.27", "ip": "10.241.164.84", "ob": "29", "device_type": null, "user_id": "17059165", "device_id": "2255508521", "operation": "LIV/channel-change", "transaction_id": "476707713_2255508521_29", "status": null, "timestamp": "2021-05-11T20:52:11.852-03:00", "bit_rate": "7Mbps", "qoe_sessionId": null, "type_event": null};;;;;N/A;17059165;;10.241.164.84;IPv4;29;HD;;;;;;;;;WIFI;DTV;IPTV;2255508521;0;4D535443;BA-ICS42-GPON;CEICS02;SCSJS01;RCSJS01;;;;;;0;0;0;0;0;0;0;0;3.0.11;1280x720@60p;N/A;2020.06.26.00.08.27;5263162195;LiveTV;N/A;;;1;ALL;VIP4242W;239.130.1.0:22220;0;0;;;EBVS;;

Which in this case would be 30 and 31, but I can't find how to update that pair of fields within the file. Any suggestion?

sed awk unix linux
1 Answers

You can't directly edit a bz2 compressed file, the only way of achieving what you're asking for that I can think of is this:

bzcat OBACountryIPTV_CALICUX_Report_Session_Local_Timezone_2021-07-02_000000.csv.bz2 | awk 'BEGIN{FS=OFS=";"}{$30=gensub(/^..(..)\/(..)\/(..)(.*)$/, "\\3/\\2/\\1\\4","1",$30);$31=gensub(/^..(..)\/(..)\/(..)(.*)$/, "\\3/\\2/\\1\\4","1",$31);print }' | bzip2 > OBACountryIPTV_CALICUX_Report_Session_Local_Timezone_2021-07-02_000000.csv.bz2.tmp
mv OBACountryIPTV_OBACountryIPTV_CALICUX_Report_Session_Local_Timezone_2021-07-02_000000.csv.bz2.tmp CALICUX_Report_Session_Local_Timezone_2021-07-02_000000.csv.bz2  #if the size looks right and you verified the content has changed as expected.

I'm certain that there are easier/more efficient ways of doing the rearranging with awk; also note that gensub is a GNU awk extension and not POSIX-compliant.

Explanation of the awk part:

BEGIN{FS=OFS=";"} - tell awk that ; is a field separator.

$30=gensub(/^..(..)\/(..)\/(..)(.*)$/, "\\3/\\2/\\1\\4","1",$30) - this parses the 30th field (the first of the unwanted date-format fields), ^.. matches the century section of the year, (..) captures the year and decade, the subsequent ones capture month and day, respectively. (.*) then captures the time. "\\3/\\2/\\1\\4" reconstructs the time and date using the captured groups (they get numbered in the order they're captured, so \\1 matches the last two digits of the year) and reassigns the value to the 30th field.

We then just repeat the same process for the 31st field.

user_13947290
Related