Shuffling parts of a file between markers

user_6715227 0
Viewed 76

I have files that look like this:

asdasadsdasdas
dasdasdasdasd
asdas
dasd
asdas
das
das
das
das
das
#SHUFFLE_MARK_START
das
d
das
das
dasd
asd
asdas
das
das
afs
sf
#SHUFFLE_MARK_END
fas
fas
fas
fas
fas
fas
fas
fas

I want to shuffle only part of file between two markers - #SHUFFLE_MARK_START and #SHUFFLE_MARK_END, shuffle markers can be any string I want, they just need to be unique filewise, any ideas how to do it in bash so it's efficient?

I already tried to do it by doing something like

cat file | grep -P '.+#SHUFFLE_MARK_START' > start
cat file | grep -P '#SHUFFLE_MARK_START.+#SHUFFLE_MARK_FINISH' | shuff | > middle
cat file | grep -P '#SHUFFLE_MARK_FINISH.+' > end
echo start middle end > shuffled

but it's pretty slow for big files, commands above may be slightly wrong as I'm typing them from memory just to show idea

shuffle awk bash
3 Answers

Here's a perl script that does it:

#!/usr/bin/perl
# Usage: foo.pl input.txt > output.txt
# or
# foo.pl < input.txt > output.txt
use warnings;
use strict;
use List::Util qw/shuffle/;
my $in_block = 0;
my @lines;
while (<>) {
  if (/#SHUFFLE_MARK_START/) {
    print;
    $in_block = 1;
  } elsif (/#SHUFFLE_MARK_END/) {
    print shuffle(@lines);
    print;
    $in_block = 0;
    @lines = ();
  } elsif ($in_block == 0) {
    print;
  } else {
    push @lines, $_;
  }
}

(If you don't want to include the #SHUFFLE_MARK_START etc lines, remove the appropriate print; lines)

user_99521960

awk to the rescue! assumes unique markers

$ awk '/#SHUFFLE_MARK_END/   {c++} 
                             {print > (FILENAME "." c+0)} 
       /#SHUFFLE_MARK_START/ {c++}' file
$ cat file.0 <(shuf file.1) file.2 > file.shuffled

difficult to see in the random file, here is a test script as well

$ seq 20 | sed 's/11/#SHUFFLE_MARK_START/;s/16/#SHUFFLE_MARK_END/' > file
$ awk ...
$ cat file.0 <(shuf file.1) file.2

1
2
3
4
5
6
7
8
9
10
#SHUFFLE_MARK_START
14
15
13
12
#SHUFFLE_MARK_END
17
18
19
20

UPDATE

here is a combined awk script, without intermediate files

$ awk '/#SHUFFLE_MARK_END/   {c++; close("shuf")} 
                             {if(c%2) print | "shuf"; else print}
       /#SHUFFLE_MARK_START/ {c++}' file

which will handle multiple (non-intersecting) sections

for this test file

$ seq 20 | sed -E 's/1?3/#SHUFFLE_MARK_START/;s/1?7/#SHUFFLE_MARK_END/' > file

$ awk ... file

1
2
#SHUFFLE_MARK_START
5
6
4
#SHUFFLE_MARK_END
8
9
10
11
12
#SHUFFLE_MARK_START
14
16
15
#SHUFFLE_MARK_END
18
19
20
user_14358690

With GNU awk for co-processes and the UNIX "shuf" command:

$ cat tst.awk
BEGIN { shuf="shuf" }
/^#SHUFFLE_MARK_END/ {
    close(shuf, "to")
    while ( (shuf |& getline line) > 0 ) {
        print line
    }
    close(shuf)
    inShuf=0
}
inShuf  { print |& shuf }
!inShuf { print }
/^#SHUFFLE_MARK_START/ { inShuf=1 }

.

$ awk -f tst.awk file
asdasadsdasdas
dasdasdasdasd
asdas
dasd
asdas
das
das
das
das
das
#SHUFFLE_MARK_START
sf
das
asdas
dasd
das
d
das
das
afs
das
asd
#SHUFFLE_MARK_END
fas
fas
fas
fas
fas
fas
fas
fas
user_17450010
Related