Manage multiple clusters in Hadoop OR Distributed Computing Framework

user_1838343 0
Viewed 77

I have five computers networked together. Among them one is master computer and another four are slave computers.

Each slave computer has its own set of data (a very big integer matrix). I want to run four different clustering programs in four different slaves. Then, take the results back into the master computer for further processing (such as visualization).

I initially thought to use Hadoop. But, I cannot find any nice way to convert the above problem (specifically the output results) into the Map Reduce framework.

Is there any nice open-source distributed computing framework by using which I can perform the above task easily?

Thanks in advance.

distributed-computing machine-learning hadoop parallel-processing database
2 Answers

You should used YARN for manage multiple clusters or resources

YARN is the prerequisite for Enterprise Hadoop, providing resource management and a central platform to deliver consistent operations, security, and data governance tools across Hadoop clusters.

Reference

user_74774080

It seems that you have already stored the data on each of the nodes, so you have already solved the "distributed storage" element of the problem.

Since each node's dataset is different, this isn't a parallel processing problem either.

It seems to me that you don't need Hadoop or any other big data framework. However, you can embrace the philosophy of Hadoop by taking the code to the data. You run the clustering algorithm on each node, and then handle the results in whatever way you need. A caveat would be if you also have a problem in loading the data and running the clustering algorithm on each node, but that is a different problem.

user_10098230
Related