Is it possible to share a session between Selenium and a LWP::UserAgent?

Viewed 122

I'm trying to download a file from a website which requires a login process. This process can be done manually as follows:

  1. Open the browser and login into the page
  2. If you have the url just enter it and you will access the file.

My problem is more complex that that, and I have to do some processing to get that url with selenium. But I'm not able to download the file with the common approaches:

  1. Getting the url (I get a croak which can not be escaped with try/catch).
  2. Generating a javascript element and clicking it generates an exception in the website.

So my new approach would be:

  1. Do the login stuff with selenium.
  2. Take the cookies and headers which are done in the request.
  3. Make the request with selenium.

This is how I do it:

use Selenium::Firefox;
use strict;
use warnings;
use utf8;
use Selenium::Waiter qw/wait_until/;

use LWP::UserAgent;
use File::Temp;
use HTTP::Cookies;
use Data::Dumper qw/Dumper/;


my $driver = Selenium::Firefox->new(marionette_enabled => 1 );
$driver->set_implicit_wait_timeout(3000);

$driver->get('https://example.com');

# processing and login

my $uri = 'https://example.com/myfile.tcx';

print "Downloading the following file: $url\n";

my $cookies = $driver->get_all_cookies();

# Create temporary file to download
my $tmp = File::Temp->new( TEMPLATE => 'myTemplateXXXX', SUFFIX=>'.tcx', UNLINK=>0 );

my $cookie_jar_obj = HTTP::Cookies->new(ignore_discard=>1,hide_cookie2=>1);
my $ua = LWP::UserAgent->new();
$ua->cookie_jar( $cookie_jar_obj );

foreach my $cookie_ref (@{$cookies}){
  my %cookie = %{$cookie_ref};
  my $name = $cookie{'name'};# - STRING
  my $value = $cookie{'value'};# - STRING
  my $path = $cookie{'path'};# - STRING
  my $domain = $cookie{'domain'};# - STRING
  my $secure = $cookie{'secure'};# - BOOLEAN

  print "Name: $name ";
  print "Value: $value ";
  print "Path: $path ";
  print "Domain: $domain ";
  print "Secure: $secure \n";

  $cookie_jar_obj->set_cookie(0, $name, $value
                             , $path, $domain, 443
                             , '', $secure, 14400000, 0
                    );
}



print "Cookies to be sent:\n";
print Dumper($cookie_jar_obj->get_cookies( ".myhost.com" ));

$ua->default_header('Accept' => "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
$ua->default_header('Accept-Encoding' => "gzip, deflate, br");
$ua->default_header('Accept-Language' => "en-US,en;q=0.5");
$ua->default_header('Connection' => "keep-alive");
$ua->default_header('Host' => "www.example.com");
$ua->default_header('Upgrade-Insecure-Requests' => "1");
$ua->default_header('User-Agent' => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:64.0) Gecko/20100101 Firefox/64.0");

$ua->show_progress( 1 );

my $response = $ua->get($uri, ':content_file' => $tmp->filename);

print $response->status_line,"\n" if !$response->is_success;
print $tmp->filename . "\n";

<>;
$driver->get('https://example.com');

But I still get:

401 Unauthorized (9s)

0 Answers
Related