Download URL content using PowerShell

user_240096 0
Viewed 77946

I am working in a script, where I am able to browse the web content or the 'url' but I am not able to copy the web content in it & download as a file. This is what I have made so far:

$url = "http://sp-fin/sites/arindam-sites/_layouts/xlviewer.aspx?listguid={05DA1D91-F934-4419-8AEF-B297DB81A31D}&itemid=4&DefaultItemOpen=1"
$ie=new-object -com internetexplorer.application
$ie.visible=$true
$ie.navigate($url)
while($ie.busy) {start-sleep 1}

How can I copy the content of $url and save it to local drive as a file?

Update:

I got these errors:

Exception calling "DownloadFile" with "2" argument(s): "The remote server returned an error: (401) Unauthorized." At :line:6 char:47 + (New-Object system.net.webclient).DownloadFile( <<<< "$url/download-url-content", 'save.html' )

Missing ')' in method call. At :line:6 char:68 + (New-Object system.net.webclient).DownloadFile( "$url", 'save.html' <<<<

Exception calling "DownloadFile" with "2" argument(s): "The remote server returned an error: (401) Unauthorized." At :line:6 char:47 + (New-Object system.net.webclient).DownloadFile( <<<< "$url", 'save.html' )

Ok, let me explain more, on what I am trying to do: I have a excel file in our share point site & this is the file I am trying to download locally(any format), which is a part of the script, so that for the later part of the script, I can compare this file with other data & get an output.

Now if I can somehow map "my documents" from the site & able to download the file, that will also work for me.

powershell
7 Answers

As I understand it, you try to use IE because if automatically sends your credentials (or maybe you didn't know of any other option).

Why the above answers don't work is because you try to download file from SharePoint and you send an unauthenticated request. The response is 401.

This works:

PS>$wc=new-object system.net.webclient
PS>$wc.UseDefaultCredentials = $true
PS>$wc.downloadfile("your_url","your_file")

if the the current user of Posh has rights to download the file (is the same as the logged one in IE).

If not, try this:

PS>$wc=new-object system.net.webclient
PS>$wc.Credentials = Get-Credential
PS>$wc.downloadfile("your_url","your_file")
user_752240

If you just want to download web content, use

(New-Object System.Net.WebClient).DownloadFile( 'download url content', 'save.html' )

Jay_Bazuzi_53140

I'm not aware of any way to save using that interface.

Does this render the page properly:

PS>$wc=new-object system.net.webclient
PS>$wc.downloadfile("your_url","your_file")
Marco_Shaw_264780

If you're truly only concerned with the raw string content, the best route, as mentioned by a few others, is using the constructs within .NET to do this. However, I think in the previous answers a few opportunities are missed.

  • It's often best to use WebRequest over WebClient as it provides better control over the entire request cycle
  • Response buffering via System.IO.StreamReader, made possible by using WebRequest
  • Creating a testable, reusable tool. Which is the very nature and purpose of PowerShell
function Get-UrlContent {
    <#
    .SYNOPSIS
        High performance url fetch

    .DESCRIPTION
        Given a url, will return raw content as string.

        Uses: 
        System.Net.HttpRequest
        System.IO.Stream
        System.IO.StreamReader

    .PARAMETER Url
        Defines the url to download

    .OUTPUTS
        System.String

    .EXAMPLE
        PS C:\> Get-UrlContent "https://www.google.com"
        "<!doctype html>..."
    #>

    [cmdletbinding()]
    [OutputType([String])]
    param(
        [Parameter(Mandatory, ValueFromPipeline)]
        [ValidateNotNullOrEmpty()]
        [string] $Url)

    Write-Debug "`n----- [Get-UrlContent]`n$url`n------`n`n"

    $req = [System.Net.WebRequest]::CreateHttp($url)

    try {
        $resp = $req.GetResponse()    
    }
    catch {
        Write-Debug "`n------ [Get-UrlContent]`nDownload failed: $url`n------`n"
    }
    finally {
        if ($resp) {
            $st = $resp.GetResponseStream()
            $rd = [System.IO.StreamReader]$st

            $rd.ReadToEnd()     
        }

        if ($rd) { $rd.Close() }
        if ($st) { $st.Close() }
        if ($resp) { $resp.Close() }   
    }
}
user_24212770
Related