I have a large dataset for analysis and I am looking for shell scripting to filter the rows only to what I require, so I am able to load the dataset for further analysis in R.
The structure for data is as follows:
Size,ModifiedTime,AccessTime,contentid
4886,"Jun 11, 2009 06:51:08 PM","Mar 15, 2013 09:24:53 AM",000000285b7925f511b3159a72f80a4a
4096,"Aug 21, 2008 03:54:28 PM","May 12, 2009 04:45:41 PM",0000011afae4d1227c4df57b410ea52c
84848,"Feb 12, 2007 12:40:00 PM","Apr 07, 2014 09:39:03 AM",000001cec02017ca3eb81ddc4cd1c9ff
518,"Aug 22, 2006 02:12:03 PM","Dec 25, 2007 06:48:18 AM",00000233565d1c17c3135a9504c455ca
264158,"Dec 08, 2009 03:28:14 PM","Apr 08, 2013 11:52:15 AM",000003020ba74b9d1b6075d3c1b8fcb3
725963,"Sep 29, 2008 03:45:21 PM","May 17, 2011 08:48:40 AM",0000034b98d29d84ce7b61ee68be7658
1340,"Sep 07, 2011 03:36:54 AM","Mar 12, 2013 02:55:01 AM",000004ed899e26ae1c9b1ece35a98af1
75264,"Jul 28, 2011 05:09:58 PM","Jun 07, 2014 04:21:28 PM",000005a09fd2eb706c5800eb06084160
198724,"Jul 23, 2012 02:25:58 PM","Jan 21, 2013 12:58:07 PM",0000060b9d552c35f281b5033dcfa1b4
It is essentially a large csv file.
Now I want to filter rows for which AccessTime is less than 10 years and then write it into a separate csv file, which in this case should print 2nd row (excluding header)
I tried the following: create a temp time variable and compare with the AccessTime, if it's less then print row.
BEGIN{
FPAT = "([^,]+)|(\"[^\"]+\")"; #this to read csv as some column value contains ,
OFS=",";
date=$(date -d "-3650 days" +"%s"); #temp time variable in epoch format
}
{
command="date -d" $6 " +%s"; #$6 refers to AccessTime column
( command | getline temp ); #converts Accesstime value to epoch format
close(command);
if(temp<date) print $6
}
But when I run this command, it doesn't print anything. Any help is much appreciated.
Desired output:
Size,ModifiedTime,AccessTime,contentid
4096,"Aug 21, 2008 03:54:28 PM","May 12, 2009 04:45:41 PM",0000011afae4d1227c4df57b410ea52c
518,"Aug 22, 2006 02:12:03 PM","Dec 25, 2007 06:48:18 AM",00000233565d1c17c3135a9504c455ca