Directory.Move() throws IOException regardless of the FileShare options for internal files

I am trying to rename or move the folder with a file, which is located in this folder being opened for reading. This blocks the folder renames, even though the file is opened with FileShare.Delete, which to my understanding, should allow rename.

Here is my C# code that reads the file:

using (FileStream fs = File.Open(@"C:\Test\1.txt", FileMode.OpenOrCreate, FileAccess.Read, FileShare.ReadWrite | FileShare.Delete))
{
    // My long-running code here.
}

Here is my code that renames the "C:\Test" folder:

Directory.Move(@"C:\Test", @"C:\Test1");

It throws:

IOException: 'The process cannot access the file because it is being used by another process.'

I do not have any Windows Explorer open or any other apps blocking the file or folder.

At the same time "create folder -> move file -> delete folder" sequence works just fine:

    Directory.CreateDirectory(@"C:\Test1");
    File.Move(@"C:\Test\1.txt", @"C:\Test1\1.txt");
    Directory.Delete(@"C:\Test");

How is this possible? How can I rename the folder with the file being opened?