500 Can't connect to url with lwp in perl

Question

I'm trying to parse some json data with the fandom wikia API. When I browse to my marvel.fandom.com/api request I get following JSON output: {"batchcomplete":"","query":{"pages":{"45910":{"pageid":45910,"ns":0,"title":"Uncanny X-Men Vol 1 171"}}}}

Nothing to fancy to begin with and running it through a JSON parser online gives following output:

{
   "batchcomplete":"",
   "query":{
      "pages":{
         "45910":{
            "pageid":45910,
            "ns":0,
            "title":"Uncanny X-Men Vol 1 171"
         }
      }
   }
}

which seems to be ok as far as I can see

I want to get the pageid for several other requests but I can't seem to get the same output through Perl. The script:

#!/usr/bin/perl
use strict;
use warnings;
use LWP::Simple;
use JSON;
use Data::Dumper;

my $url = "https://marvel.fandom.com/api.php?action=query&titles=Uncanny%20X-Men%20Vol%201%20171&format=json";

my $json = getprint( $url);
die "Could not get $url!" unless defined $json;

my $decoded_json = decode_json($json);
print Dumper($decoded_json);

but this gives following error:

Could not get https://marvel.fandom.com/api.php?action=query&titles=Uncanny%20X-Men%20Vol%201%20171&format=json! at ./marvelScraper.pl line 11.

When I change the get to getprint for some extra info, I get this:

500 Can't connect to marvel.fandom.com:443 

malformed JSON string, neither tag, array, object, number, string or atom, at character offset 0 (before "(end of string)") at ./script.pl line 13.

I tried this on another computer and still get the same errors.

The versions of LWP::Simple and LWP::Protocol::https

/usr/bin/perl -MLWP::Simple -E'say $LWP::Simple::VERSION'
6.15
/usr/bin/perl -MLWP::Protocol::https -E'say $LWP::Protocol::https::VERSION'
6.09

Appearantly it has something to do with the Bash Ubuntu on Windows since on a Ubuntu 18.04 I get (with the same script) following response:

JSON text must be an object or array (but found number, string, true, false or null, use allow_nonref to allow this) at ./test.pl line 13.
{"batchcomplete":"","query":{"pages":{"45910":{"pageid":45910,"ns":0,"title":"Uncanny X-Men Vol 1 171"}}}}

user_1620937 · Answer

Actually, the very same script works from my Bash Ubuntu on Windows with the get() command instead of the getprint() you gave after editing your question.

orabig@Windows:~/DEV$ ./so.pl
$VAR1 = {
          'query' => {
                       'pages' => {
                                    '45910' => {
                                                 'pageid' => 45910,
                                                 'ns' => 0,
                                                 'title' => 'Uncanny X-Men Vol 1 171'
                                               }
                                  }
                     },
          'batchcomplete' => ''
        };

So maybe you have another issue that has nothing to do with Perl or Ubuntu.

Can you try this for example ?

curl -v 'https://marvel.fandom.com/api.php?action=query&titles=Uncanny%20X-Men%20Vol%201%20171&format=json'

Maybe you just hit the site too much, and the 500 error is just a result of some anti-leech protection ?

500 Can't connect to url with lwp in perl

1 Answers