Request multithreading with array

Henrique 0
Viewed 53

This script reads a file of urls to do multithreading HTTP requests.

How can I use an array with urls to make multithreading requests?

My array will have something like:

@array = ("https://example.com/xsd","https://example.com/xys","https://example.com/des","https://example.com/hduei");

I need to remove the function of reading file with urls, but I can not.

#!/usr/bin/perl
use strict;
use warnings;
use threads;
use threads::shared;

use Fcntl              qw( LOCK_EX );
use IO::Handle         qw( );
use LWP::UserAgent     qw( );
use Thread::Queue 3.01 qw( );

use constant NUM_WORKERS => 20;

my $output_lock :shared;
my $output_fh;
sub write_to_output_file {
   lock($output_lock);
   print($output_fh @_);
   $output_fh->flush();
}

sub worker {
   my ($ua, $url) = @_;
   my $response = $ua->get($url);
   write_to_output_file("$url\n")
      if $response->success
      && $response->content =~ /Exist/;       
}

{
   $output_fh = \*STDOUT;  # Or open a file.

   my $q = Thread::Queue->new();
   for (1..NUM_WORKERS) {
      async {
         my $ua = LWP::UserAgent->new( timeout => 15 );
         while (my $job = $q->dequeue()) {
            worker($ua, $job);
         }
      };
   }

   while (<>) {
      chomp;
      $q->enqueue($_);
   }

   $q->end();
   $_->join() for threads->list();
}
perl multithreading request http
1 Answers

This part of the code you've shown reads filenames from the command line arguments, and then reads all the lines in these files. It then iterates over the lines.

   while (<>) { # <--- here
      chomp;
      $q->enqueue($_);
   }

You can replace that with an array, which of course needs a for loop. Make sure to remove the chomp, as there won't be a need to remove newlines.

   foreach (@urls) {
      $q->enqueue($_);
   }
simbabque0
Related