Powershell script stops working when ran through task scheduler

I am running the script below as a scheduled task with the user logged on the server. It converts an xls file to csv using the Excel.Application COM object. The conversion works, but eventually breaks and I don't know why.

I have the task run the following command which should in theory allow it to run constantly:

powershell.exe -noexit -file "filename.ps1"

Any thoughts on what to try?

$server = "\\server"
$xls = "\path\XLS\"
$csv = "\path\CSV\"
$folder = $server + $xls
$destination = $server + $csv

$filter = "*.xls" # <-- set this according to your requirements

$fsw = New-Object IO.FileSystemWatcher $folder, $filter -Property @{
    IncludeSubdirectories = $true # <-- set this according to your requirements
    NotifyFilter = [IO.NotifyFilters]"FileName, LastWrite"
}

Register-ObjectEvent $fsw Created -SourceIdentifier FileCreated -Action {

    $path = $Event.SourceEventArgs.FullPath
    $name = $Event.SourceEventArgs.Name
    $changeType = $Event.SourceEventArgs.ChangeType
    $timeStamp = $Event.TimeGenerated

    $excelFile = $folder + $name
    $E = New-Object -ComObject Excel.Application
    $E.Visible = $false
    $E.DisplayAlerts = $false
    $wb = $E.Workbooks.Open($excelFile)

    foreach ($ws in $wb.Worksheets) {
        $n = "output_" + $name -replace ".XLS"
        $ws.SaveAs($destination + $n + ".csv", 6)
    }

    $E.Quit()     
}