Dealing with binary streams read from the web

user_4400590 0
Viewed 98

I use wget to read a page from the web. But sometimes I get gzipped binary stream instead of plain text html file. What is the best way to decide if the data I get is binary or plain text? If I try to match the data with letter or number (text), I simply get "Malformed UTF-8".

my $result = run << wget -k -q -O $aPage "$aURL" >>, :err;

I need to know if $result is binary (gzip) or plain text.

if $result ~~ / <:L + :N> / { } # this will fail with "Malformed UTF-8" if $result is a binary stream

***** Is there a Raku package to get a plain text html page source from ANY url?

Thanks.

raku binary html
1 Answers

What is the best way to decide if the data I get is binary or plain text?

Perhaps the most general answer is perhaps to turn to the Raku module Data::TextOrBinary, which can be used to determine if some data is likely to be text or binary; it's a heuristic approach.

The best way to determine the nature of a HTTP response is to look at the Content-type header. Rather than calling out to wget, try one of the various Raku HTTP modules, which can be installed with the zef package manager. For example, you can use Cro::HTTP::Client:

use Cro::HTTP::Client;
my $response = await $aURL;
say "Content is " ~ $response.content-type;
my $body = await $response.body;
say $body ~~ Blob ?? "It's binary!" !! "It's not binary!";

If I try to match the data with letter or number (text), I simply get "Malformed UTF-8".

By default run assumes that is what it will get. You can also pass the :bin argument to run, in order to get a Blob (any binary data) rather than having it to to decode it automatically. However, you'll still need to decide what to do in the event it's not textual data.

user_78325840
Related