So, I've been looking at Hadoop with keen interest, and to be honest I'm fascinated, things don't get much cooler.
My only minor issue is I'm a C# developer and it's in Java.
It's not that I don't understand the Java as much as I'm looking for the Hadoop.net or NHadoop or the .NET project that embraces the Google MapReduce approach. Does anyone know of one?
Have you looked at using Hadoop's streaming?
I use it in python all the time :-).
I'm starting to see that the heterogeneous approach is often the best and it looks like other folks are doing the same.
If you look at projects like protocol-buffers or facebook's thrift you see that sometimes it's just best to use an app written in another language and build the glue in the language of your preference.
See http://research.microsoft.com/en-us/projects/dryadlinq/default.aspx or http://msdn.microsoft.com/en-us/library/dd179423.aspx
I would say that DryadLinq is the closest thing that us .NET folk have to Hadoop. But it depends what you want to use hadoop for. If you are looking for the optimized self maintaining distributed file (DFS) system then DryadLINQ isn't what you are looking for. It has an analog to the DFS but you have to manually build the partitions and distribute each partition.
That being said, if its the distributed execution aspect of Hadoop that you are looking for than DryadLINQ is truly wonderful (and no, i'm not affiliated with MS). As long as you have a Microsoft HPC cluster setup than getting going with DryadLINQ is really easy.
The code you write is really just straight LINQ code, except instead of executing the LINQ on IEnumerable<T> you have to execute it on PartitionedTable<T> (the self build distributed data structure).
What has really been cool about DryadLINQ is the fast turn around time (try, test, adjust, repeat) when developing algorithms. You just write LINQ code to do your calculations and DryadLINQ will take care of the whole distributed execution part. It's the most natural analog I've come across that makes writing code for distributed processing just like writing code for single process processing.
You can look into something like RavenDb it provides very decent support for MapReduce for a fairly large size of data. as it is built in .Net so a proper LINQ client API is available.
To get you started you can read my blog entery.
