Payara/Hazelcast Instances Object Hierarchy

As very one know, Payara/Hazelcast application platform not well documented with their Architecture level Objects relationship or hierarchy. I did not find any thing in their documents or on the Internet.

I'm trying to find out away to understand Payara/Hazelcast objects (domain, cluster, deployment group, Hazelcast Instance, applications) relationship.

My environment as: Payara 5.2021.3 community edtion runs on two od identical machines with Centos 8 Linux, it was configured as:

- domain name: production
- cluster name: thss
- nodes(two): localhost-production(CONFIG), centos8thssv2-production(SSH)
- instances(two): ThssEJBServer (runs on localhost-production), ThssWEBServer (runs on centos8thssv2-production)

I'm trying to get instances information form 4 of different ways as:

1. domain log file show:

   Payara Data Grid State: DG Version: 4 DG Name: thss DG Size: 3 Instances: { DataGrid: thss Name: ThssWEBServer Lite: false This: false UUID: 7d51a149-ce9f-43ce-a905-22a63a685689 Address: /10.0.1.152:5900 DataGrid: thss Name: server Lite: false This: false UUID: 796ce6ef-8694-44cb-9378-1724dfd3556c Address: /10.0.1.102:4900 DataGrid: thss Name: ThssEJBServer Lite: false This: true UUID: cf5fb684-2729-4f4e-91ef-cbaf85e0ddfa Address: /10.0.1.102:5900 }

   There are three Hazelcast instances names as: server(DAS), other two were added to the domain

2. admin_ui:

   

3. asadmin CLI:

   asadmin> list-instances ThssEJBServer running
   ThssWEBServer running
   Command list-instances executed successfully. asadmin>

4. programmatically

    Set<HazelcastInstance> instanceSet = Hazelcast.getAllHazelcastInstances();
    logger.info("-------------------There are "+instanceSet.size()+" found in the cluster");
    if(instanceSet.size()>0){
        for (HazelcastInstance instance : instanceSet) {
            logger.info("Instance name: "+instance.getName());
        }
    }
   

This code was put to EJB module and annotate it as @Startup and @Singleton, after deploy it onto Instance ThssEJBServer, the ThssEJBServer log shows:

[2021-05-17T15:55:23.645+1000] [Payara 5.2021.3] [INFO] [] [com.longz.thss.ejb.listener.ThssHzProvider] [tid: _ThreadID=118 _ThreadName=admin-thread-pool::admin-listener(1)] [timeMillis: 1621230923645] [levelValue: 800] [[
-------------------There are 1 found in the cluster]]

[2021-05-17T15:55:23.645+1000] [Payara 5.2021.3] [INFO] [] [com.longz.thss.ejb.listener.ThssHzProvider] [tid: _ThreadID=118 _ThreadName=admin-thread-pool::admin-listener(1)] [timeMillis: 1621230923645] [levelValue: 800] [[
Instance name: magical_montalcini]]

please note, there is only one instance found and its name as: magical_montalcini and this name is not shows up in any other instance

This is not only result, When first couple of time runs the same Java code, we find the pieces of Java code can give the right Hazelcast instances (3 of them), but after about 4-5 time re-run this code, we got upper result from log, the early result lost and not re-produceable any more.

Other funny thing is:

When I deployed the applicaion to the domain, it shows up in Admin_ui as well

but when I try to list it from asadmin-CLI, by run asadmin list-applications, there is only one show up

asadmin> list-applications<br>
__monitoringconsole  <health, metrics, openapi, web>  
Command list-applications executed successfully.<br>
asadmin>