Convert XML like format to CSV in Perl

Question

So, I have a file that looks like this:

random stuff in the beginning...

 
   bob
   7
   Great_Britain
 

 
   bob_secondbob
   7
   Great_Britain
 

 
   bam
   7
   Great_Britain

etc...

And, I want to just have it as a CSV like this:

name,age,country
bob,7,Great_Britain
bob_secondbob,7,Great_Britain
bam,7,Great_Britain

So I was wondering how would I go about doing this?

So my current code has the word like bob and bam, so i have been greping for it like this and using sed to just get the values:

grep -A4 "" $file | grep -A4 "$bob.*" | grep "" 
grep -A4 "" $file | grep -A4 "$bob.*" | grep ""
grep -A4 "" $file | grep -A4 "$bob.*" | grep ""

etc...

Where the variable $bob is "bob";

But the problem is that there are 2 instances of bob.* and I don't know how to separate them so I can print them out...

I looked at XML modules but this file isn't fully XML so I cant use that...

Like bam would be easy to print out by grepping it like that, but if there are multiple instances of bob.* where I need both of them, idk how to print it out properly because grep would return multiple results.

Any suggestion as to how i might go about this?

user_4653379 · Answer

It is clearly stated that the "file isn't fully XML" and that XML libraries can't be used. Bummer :(

Then this is parsed using regex. Remember that then one has to always keep an eye on input files to see whether their format changed; even a smallest change can easily throw off a regex, causing the program breakdown at best or, much worse, a quiet bug.^†

The shown format is easy to parse. Here's a basic take, parsing the XML-like component section for any tags and their values, then printing for a given set of actual tags in needed order.

use warnings;
use strict;
use feature 'say';

my $section_name = 'component';   # XML-like section to parse
my @tags = qw(name age country);  # given tags and their order

my (%record, $in_XML);

while (<>) {
    if    (/^\s*<$section_name>\s*$/)   { $in_XML = 1 }
    elsif (/^\s*<\/$section_name>\s*$/) { $in_XML = 0 } 
    
    if ( $in_XML and m{<([^<]+)> ([^<]+) }x ) { 
        push @{$record{$1}}, $2; 
    }   
}

# Print out CSV-style output, with given tags
say join ',', @tags;
for my $i (0..$#{$record{$tags[0]}}) { 
    say join ',', map { $record{$_}->[$i] } @tags;
}

A few assumptions are made about tags. Some important ones: each tag-pair is on one line; all tag names are unique. If these don't hold the code need be adjusted, what can be done but would need some work.

On top of matching the XML-like opening and closing tag-pair, ..., I've also added a flag for when the processing is inside a component section. Testing the flag inside the if condition allows for other processing outside of XML, otherwise we could have next if not $in_XML; before the if condition. This whole business may be unnecessary if there is no chance for an accidental XML-like tag-pair elsewhere in text.

Note that one doesn't have to specify and use @tags but can print for tags as found in the file, which are my @tags = keys %record, if that is acceptable and if order doesn't matter.

Please add tests of whether those tags and their values are indeed what one expects. Realistic input files tend to occasionally have missing or unexpected parts.

^† It'd be far better to remedy the "isn't fully XML" (make it XML) and use a library, if possible.

user_12313309 · Answer

Input data read sequentially, once we come across it indicates data follow.

Now we start parse data until we read into hash and then push result into an array.

Once whole file processed, output keys of the hash as header and values of data hash for each element by join with ,.

use strict;
use warnings;
use feature 'say';

my @data;
my @fields = qw/name age country/;

while(  ) {
    if( // ) {
        my $component;
        while(  ) {
            last if /<\/component>/;
            $component->{$1} = $2 if /^\s+<(.+?)>(.*?)@{@fields} for @data;

__DATA__


 
   bob
   7
   Great_Britain
 

 
   bob_secondbob
   7
   Great_Britain
 

 
   bam
   7
   Great_Britain

Second variant with match range operator and named capture groups

use strict;
use warnings;
use feature 'say';

my(@data,%component);
my @fields = qw/name age country/;

while(  ) {
    chomp;
    if( //../<\/component>/ ) {
        $component{$+{tag}} = $+{val} if m!<(?.+?)>(?.*?)!;
        if( /<\/component>/ ) {
            my %hash = %component;
            push @data, \%hash;
            %component = ();        
        }
    }
}

say join ',', @fields;
say join ',', $_->@{@fields} for @data;

__DATA__


 
   bob
   7
   Great_Britain
 

 
   bob_secondbob
   7
   Great_Britain
 

 
   bam
   7
   Great_Britain

Output

name,age,country
bob,7,Great_Britain
bob_secondbob,7,Great_Britain
bam,7,Great_Britain

user_589924 · Answer

You should still use a proper XML parser. It will be faster, simpler, less error-prone, more readable, more maintainable, and more. Simply extract the XML bits from the file and use the normal approach from there.

For all the same reasons, use a proper CSV generator instead of rolling out your own.

use Text::CSV_XS qw( );
use XML::LibXML  qw( );

my $xml_parser = XML::LibXML->new()
my $csv = Text::CSV_XS->new({ auto_diag => 2, binary => 1 });

$csv->say(\*STDOUT, [qw( name age country )]);

my $in_xml = 0;
my $xml;
while (
   $in_xml ||= /parse_string($xml);
      my $name    = $doc->findvalue("/component/name");
      my $age     = $doc->findvalue("/component/age");
      my $country = $doc->findvalue("/component/country");
      $csv->say(\*STDOUT, [ $name, $age, $country ]);

      $in_xml = 0;
      $xml = undef;
   }
}

Alternative way to parse the XML:

my $doc = $xml_parser->parse_string($xml);
my %rec;
$rec{ $_->nodeName } = $_->textContent()
   for $doc->findnodes("/component/*");

$csv->say(\*STDOUT, [ @rec{qw( name age country )} ]);

Convert XML like format to CSV in Perl

3 Answers