How to download files that contain a pound/hash sign '#' in the filename from an FTP server using PowerShell's FtpWebRequest class

I have built a script for downloading files from an FTP server. The script works with all files I have attempted to download except files that contain a #. After doing some research I am unable to find a solution to this problem. My code for downloading a file is listed below.

function Get-FtpFile
{
  Param ([string]$fileUrl, $credentials, [string]$destination)
  try
  {
    $FTPRequest = [System.Net.FtpWebRequest]::Create($fileUrl)
    if ($credentials) 
    {
        $FTPRequest.Credentials = $credentials
    }
    $FTPRequest.Method = [System.Net.WebRequestMethods+Ftp]::DownloadFile
    $FTPRequest.UseBinary = $true

    # Send the ftp request
    $FTPResponse = $FTPRequest.GetResponse()

    # Get a download stream from the server response
    $ResponseStream = $FTPResponse.GetResponseStream()

    # Create the target file on the local system and the download buffer
    $LocalFile = New-Object IO.FileStream ($destination,[IO.FileMode]::Create)
    [byte[]]$ReadBuffer = New-Object byte[] 1024

    # Loop through the download
    do {
        $ReadLength = $ResponseStream.Read($ReadBuffer,0,1024)
        $LocalFile.Write($ReadBuffer,0,$ReadLength)
       }
    while ($ReadLength -ne 0)
    $LocalFile.Close()
  }
  catch [Net.WebException]
  {
    return "Unable to download because: $($_.exception)"
  }
}

I have tried using WebRequest.DownloadFile() instead and it still does not work with files that contain a #, I have also tried renaming the file using the FtpWebRequest rename method and that also did not work.

Does anyone know of any solution or workaround to this problem?