How to properly delete a file in .Net Core with C# from a folder with large amount of other files or folders

user_1346161 0
Viewed 1317

I am removing files in a loop with File.Delete(String) in a simplest .Net Core Console application:

1. foreach (var listing in listings)
2.    File.Delete(listing.Path);

BUT the issue is that the application randomly hangs on line 2, no exception is thrown. After application restart, several files are removed successfully and the app randomly hangs, again, on another file.

There are some items to note:

  • listing.Path is an absolute path to a file on a disk
  • listing.Path looks like D:\Output\00000001\6b1c8e6c-7d83-481a-a7db-aac9024059c4.png
  • the length of listings array is about 10 items
  • file DOES exist in every case
  • Storage: Seagate Exos 7E8 512E 8TB 7200rpm 256MB ST8000NM000A 3.5" SATA III HDD
  • OS: Windows 10 64x
  • total size of Output folder is 210GB
  • Output folder contains 137 000 other folders with 928 000 files in total (approximately 5 - 10 files per folder)
  • average file size is about 350KB

I suspect that issue is somehow related to the amount of files. But File.Delete should be the most reliable way to remove a file, and it just hangs without any response for multiple hours. Which forces me to kill the process of the app. Is there a better or more efficient way to remove a file?

Update: After moving all the files to another Windows machine in a network with an SSD installed and executing the application there it worked perfectly fine. So it may look like a hardware issue. Therefore I will rephrase the question title from How to properly delete a file in .Net Core with C# to How to properly delete a file in .Net Core with C# from a folder with large amount of other files or folders

.net-core file .net c#
1 Answers

1) Don't expect to create an array with hundreds of thousands of filenames and then loop through them deleting them very fast

2) The file system, especially if the directory is on a network share, mapped drive, or in use by another application will be slow and block operations. For example, a processing adding new files to a directory as fast as possible will conflict and possibly deadlock with one deleting files as fast as possible from the same directory.

3) Files may be locked by another process

Try some tests a) How long does it take to get a list of all of the files to be deleted? b) How long does it take to delete 1000 files in a row?

One queue system I developed had large numbers of files with multiple processes creating files, multiple processes processing files and then deleting the files.

We had 3 directories: Inbound, BeingProcessed, and ForDeletion

Each new file was put in the Inbound directory Another process would move the file from the inbound directory to the BeingProcessed directory, open and lock the file, then process the file and the move the file to the ForDeletion directory Another process would delete files in the ForDeletion directory

user_130489910
Related