Is there an async version of DirectoryInfo.GetFiles / Directory.GetDirectories in dotNet?

user_73046 0
Viewed 28040

Is there an asynchronous version of DirectoryInfo.GetFiles / Directory.GetDirectories in dotNet? I'd like to use them in an F# async block, and it'd be nice to have a version that can be called with AsyncCallbacks.

Problem is I'm trying to suck in a bunch of directories, probably on SMB mounts over slow network connections, and I don't want a bunch of thread pool threads sitting around waiting for network reads when they could be doing other work.

async-workflow directoryinfo asynchronous f# .net
7 Answers

No, I don't think there is. The pool thread approach is probably the most pragmatic. Alternatively, I guess you could drop down to P/Invoke - but that would be a lot more work.

Marc_Gravell_233540

I didn't find an async version of GetFiles, however if you look at the sourcecode for other Async operations, they're defined as follows:

module FileExtensions =

        let UnblockViaNewThread f =
            async { //let ctxt = System.Threading.SynchronizationContext.Current
                    do! Async.SwitchToNewThread ()
                    let res = f()
                    do! Async.SwitchToThreadPool ()
                    //do! Async.SwitchTo ctxt
                    return res }

        type System.IO.File with
            static member AsyncOpenText(path)   = UnblockViaNewThread (fun () -> System.IO.File.OpenText(path))
            static member AsyncAppendText(path) = UnblockViaNewThread (fun () -> System.IO.File.AppendText(path))
            static member AsyncOpenRead(path)   = UnblockViaNewThread (fun () -> System.IO.File.OpenRead(path))
            static member AsyncOpenWrite(path)  = UnblockViaNewThread (fun () -> System.IO.File.OpenWrite(path))
            static member AsyncOpen(path,mode,?access,?share) =
                let access = match access with Some v -> v | None -> System.IO.FileAccess.ReadWrite
                let share = match share with Some v -> v | None -> System.IO.FileShare.None
                UnblockViaNewThread (fun () -> System.IO.File.Open(path,mode,access,share))

            static member OpenTextAsync(path)   = System.IO.File.AsyncOpenText(path)
            static member AppendTextAsync(path) = System.IO.File.AsyncAppendText(path)
            static member OpenReadAsync(path)   = System.IO.File.AsyncOpenRead(path)
            static member OpenWriteAsync(path)  = System.IO.File.AsyncOpenWrite(path)
            static member OpenAsync(path,mode,?access,?share) = System.IO.File.AsyncOpen(path, mode, ?access=access, ?share=share)

In other words, the Async file, streamreader, and WebClient operations are just wrappers around the syncronous operations, so you should be able to write your own wrapper around GetFiles/GetDirectories as follows:

module IOExtensions =
    type System.IO.Directory with
        static member AsyncGetFiles(directory) = async { return System.IO.Directory.GetFiles(directory) }
        static member AsyncGetDirectories(path) = async { return System.IO.Directory.GetDirectories(path) }
Princess_405160

I've used several times this approach to get Async objects from functions/procedures, and it always worked great:


let AsyncGetDirectories path = 
    let fn = new Func<_, _>(System.IO.Directory.GetDirectories)
    Async.BuildPrimitive(path, fn.BeginInvoke, fn.EndInvoke)
user_870790

Princess's answer is the way to go for adding granularity between tasks - so this kind of thing would let other players use the thread pool:

let! x = OpenTextAsync("whatever");
// opening for something else to run
let! x = OpenTextAsync("whatever");
// opening for something else to run
let! x = OpenTextAsync("whatever");

It doesn't help as much when each one of those blocking calls is heavy - and a GetFiles over SMB is pretty much the definition of heavy.

I was hoping there was some sort of equivalent for BeginRead/EndRead for directories, and that GetFiles/GetDirectories was just a nice wrapper around lower-level calls that exposed some async variants. Something like BeginReadDir/EndReadDir.

user_730460
Related