Problem with Quotes while using variables with "&" and "%"

The problem boils down to this:

• The requirement to support not just & but also % characters in your URLs significantly complicates the solution, unfortunately, because cmd.exe (inappropriately) subjects a parameter's verbatim value passed to a subroutine via call to string interpolation, which necessitates doubling the % in the value in order to preserve them.

  • Thus, a URL read from a file with for /f must have its % programmatically doubled before passing it as a parameter to call :do_download, which is a nontrivial undertaking - see the code below and the source-code comments.

  • Note that additional effort would be required if your URL also contained ! characters.

• In your :do_download subroutine, use set "url=%~1". This stores the unquoted value of %1 (the first parameter) in variable %url%

  • Do not use set "url=%1" or set url=%~1, both of which predictably break with a %1 value that has embedded surrounding double quotes and contains &.

  • Do not use %1:"=, which doesn't do what you think it does. %1 is a parameter, whereas the string-substitution syntax you're trying to use only works with variables (e.g., %url:"=%)

• Then you can use %url% as-is inside the "..." string that forms the powershell -c argument - or you could just use %~1 directly.

  • With the intermediate %url% variable, however, you could alternatively take advantage of the fact that all cmd.exe variables (but not parameters such as %1) are also environment variables, which therefore allows PowerShell to access the URL via an environment-variable reference instead of embedding it directly in the command string (the equivalent of cmd.exe's %url% is $env:url in PowerShell). That is, you could replace the powershell call below with the following - note how enclosing '...' quoting as part of the PowerShell command must then not be used:

    • powershell -c "Write-Output $env:url"

Here's a minimal example that shows that a URL containing both & and % characters is correctly passed through to PowerShell; everything else in your code is incidental (use of for /f, specifics of the PowerShell command):

@echo off
setlocal enabledelayedexpansion

:: Use a sample URL that contains both & and % characters.
:: Note: The % must be DOUBLED here in order to escape it.
::       The resulting string will only have ONE %, as desired and will therefore
::       literally be the following, as it would be read from a file with `for /f`: 
::          http://example.org?p=%2FTest.txt&dl=1
set "url=http://example.org?p=%%2FTest.txt&dl=1"

:: Unfortunately, using `call` requires escaping of % even in variable values, 
:: as the embedded single % are then again subject to variable expansion
:: (even though they shouldn't be).
:: Doubling % *programmatically* is non-trivial and requires a helper variable
:: as well as mixing up-front and delayed expansion.
:: The following string substitution effectively doubles the embedded % chars.
:: so that the :do_download subroutine again sees them as verbatim single ones.

set "pct=%%"
call :do_download "!url:%pct%=%%%%!"

goto :eof

:do_download
  set "url=%~1"
  powershell -c "Write-Output '%url%'"
goto :eof

The above prints verbatim http://example.org?p=%2FTest.txt&dl=1, proving that the URL was correctly passed through to PowerShell.

Does this work?

@echo off
setlocal enabledelayedexpansion

set "url=!array[2]!"

call :do_download
   
:do_download 
    powershell -c "$Url=$Env:url; [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; 
       Invoke-WebRequest $Url -OutFile 'C:\Program Files (x86)\test.zip'"

You have Delayed Explansion enabled, so you should be able to just past the variable's name and expand it later.

Does this also work?

@echo off
setlocal enabledelayedexpansion

set "url=!array[2]!"

call :do_download url
GOTO :EOF

:do_download 
    powershell -c "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; 
       Invoke-WebRequest $Env:!%~1! -OutFile 'C:\Program Files (x86)\test.zip'"

EDIT:

I'm not sure if the packs.txt file contains only URLs or other content. I placed 5 copies of the sync.luckycloud.de URL, the one mentioned in the question, in a mock packs.txt file and was able to get this code to work.

@ECHO OFF
SETLOCAL EnableDelayedExpansion
GOTO :Start

:ReadTest
    SET Count=0
    FOR /F "USEBACKQ EOL=` TOKENS=* DELIMS=`" %%U IN ("C:\Program Files (x86)\Steam\steamapps\common\packs.txt") DO (
        SET /A Count+=1
        SET url[!count!]=%%U
    )
    SET "pack_sd=!url[2]!"
GOTO :EOF

:Do_Download
    PowerShell -c "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; Invoke-WebRequest $Env:pack_sd -OutFile 'C:\Program Files (x86)\Steam\steamapps\common\test.txt'"
GOTO :EOF

:Start
    CALL :ReadTest
    CALL :Do_Download
GOTO :EOF

NOTE: I believe the characters below are not allowed as part of a URL. The FOR command first looks for DELIMS, but there should never be a back tick in the URL, so DELIMS should never be found. But if one is found, it is grabbed by DELIMS before EOL, this make EOL disabled. The default for EOL is a semicolon, which appears to be a valid character in URLs. So it is important to disable EOL, or give it a value that will never be in a URL.

"%\^`{|}<>

Final Generic Version:

ReadFile: Takes name of array for saving URLs and the path of a file containing URLs. Reads file, storing URLs in array.

Do_Downloads: Takes name of array of URLs and the path where files will be saved. URLs from array are downloaded and saved to files with name WebFile{N}.txt, where {N} is the line number the URL came from in the file containing URLs.

@ECHO OFF
SETLOCAL EnableDelayedExpansion
GOTO :Start

:ReadFile
    SET Count=0
    FOR /F "USEBACKQ EOL=` TOKENS=* DELIMS=`" %%U IN ("%~2") DO (
        SET /A Count+=1
        SET %~1[!Count!]=%%U
    )
GOTO :EOF

:Do_Downloads
    FOR /L %%I IN (1, 1, %Count%) DO (
        SET Url=!%~1[%%I]!
        PowerShell -c "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; Invoke-WebRequest $Env:Url -OutFile '%~2\WebFile%%I.txt'"
    )
GOTO :EOF

:Start
    CALL :ReadFile Urls "C:\Program Files (x86)\Steam\steamapps\common\packs.txt"
    CALL :Do_Downloads Urls "C:\Program Files (x86)\Steam\steamapps\common"
GOTO :EOF