What is the best way to move half a million files to another folder based on a list?

Viewed 166

How do I move a large number of files to another folder quickest?

I have a repository of 800,000+ xml files from the IRS, and I'm trying to move a subset (approx 500,000) to a new folder. So far I've created a batch file that uses a list of filenames to move them. It's working, but it's too slow. The batch file has been executing for about an hour now, but only one file has been moved into the new folder.

Is there anyway I could improve the efficiency of my batch file? Or is there a better way to do this?


@echo off
set Source=C:\Users\tenis\OneDrive\Desktop\Data_Projects\Impact_Capital\data\IRS990_2017_2016
set Target=C:\Users\tenis\OneDrive\Desktop\Data_Projects\Impact_Capital\data\IRS990_2017
set FileList=C:\Users\tenis\OneDrive\Desktop\Data_Projects\Impact_Capital\data\filenames_990_2017.txt
echo.

if not exist "%Source%" echo Source folder "%Source%" not found & goto Exit
if not exist "%FileList%" echo File list "%FileList%" not found & goto Exit
if not exist "%Target%" md "%Target%"

for /F "delims=" %%a in ('type "%FileList%"') do copy "%Source%\%%a" "%Target%"

:Exit
echo.
echo press the Space Bar to close this window.
pause > nul

1 Answers

The batch file actually worked! When I hit cntrl c in the command line, files appeared. After rerunning it, I got what I needed.

So I guess the take away from all this is hit cntrl c if you don't know what the script is doing, and have patience. =)

Related