I read many sites related to Storm's topologies design setup. But, I didn't get clarity.
In my project, I am going to processing more than a million records. So, I planned to create topologies dynamically based on internal modules. The count might be reached more than a thousand. My doubt is what is the best way to manage topologies? How many topologies can be created in a single cluster? Are there any problems with maintaining multiple topologies?
I would say, that this really depends on your machines in the cluster, so it is hard to answer that generally - this is especially true, if the cluster has heterogeneous instances.
Basically storm can handle many topologies that you can control over the CLI or the GUI.
I am currently managing those with storm list and storm kill commands. Limits should be in the RAM, storage and network connections of the single machines. To be precisely, I would predict, that the bottleneck is the JVM size of a supervisor instance. This can hold multiple workers (that each having components like bolts & spouts that are initially configured with a JVM of 256 MB), but if there are too many workers, the overall consumed JVM per supervisor will be exceeded.
