To send tasks to other members of a cluster, our implementation does:
hazelcastInstance.getExecutorService("ourname").submitToMembers(new CallableTask<>(task), members);
Our implementation uses a plugin system. When a plugin is (re)started, it will use a new ClassLoader instance, specific to that plugin.
We are experiencing problems with plugins that are issuing cluster tasks, and get reloaded. Specifically, it appears that when a plugin is reloaded on a cluster node that has once executed a task that was using classes provided by that plugin, subsequent tasks that arrive at that cluster node will cause ClassCastException ("X cannot be cast to X").
Our hypothesis: It appears as if Hazelcast is keeping a reference to the ClassLoader it used to instantiate classes for the execution of the first task, which prevents that ClassLoader to be garbage collected when we unload the plugin. That old ClassLoader appears to be used for the execution of the second task, where we would prefer/expect the new ClassLoader to be used. As there are now instances of the same class loaded by different ClassLoaders, ClassCastExceptions occur.
Is there any truth to this? How can we resolve these issues?