Neo4j cluster not starting

user_503969 0
Viewed 404

I am building a Neo4j HA cluster and have followed the steps at https://neo4j.com/docs/operations-manual/current/tutorial/highly-available-cluster/

I have 3 machines running Neo4j in a single-instance Docker container.

Neo4j works fine as a standalone server on each machine, but when I add the HA config, it fails to start.

Each machine was created from the same prebuilt image so they are configured identically except for the ha.server_id which is unique for each machine (values are 1, 2, 3).

Firewall is off and I've verified connectivity between the machines.

The Neo4j HA config is:

ha.server_id=1
ha.initial_hosts=172.0.30.110:5001,172.0.31.39:5001,172.0.32.249:5001
dbms.mode=HA

To start Neo4j I'm running the following on each machine:

/usr/bin/docker run \
--publish=7474:7474 --publish=7687:7687 --publish=5001:5001 \
--volume=/var/lib/neo4j/data:/data \
--volume=/var/lib/neo4j/logs:/logs \
--volume=/var/lib/neo4j/conf:/conf \
--name=neo4j \
neo4j:3.0-enterprise

When I run the startup command on machines 2 and 3, and sometimes on machine 1, I get an error following 'Attempting to join cluster'. Occasionally (but not all the time) when I run the command on machine 1, it realizes it is the first machine and goes ahead to initialize itself as master.

Below are the full failure and success messages. I can't figure out why only one machine (incidentally the first) is doing what it's supposed to do sometimes and not all the time. I also need to figure out how to get the other two machines to register themselves with the cluster.

  1. Why is one node sometimes launching Neo4j with clustering and sometimes not?
  2. Why are the 2nd and 3rd nodes not registering with the cluster?
  3. Where should I look next?

Error (seen on all 3 machines):

> 2017-05-30 09:18:35.502+0000 INFO  Starting... 2017-05-30
> 09:18:35.833+0000 INFO  Write transactions to database disabled
> 2017-05-30 09:18:36.116+0000 INFO  Bolt enabled on 0.0.0.0:7687.
> 2017-05-30 09:18:36.131+0000 INFO  Initiating metrics... 2017-05-30
> 09:18:36.937+0000 INFO  Attempting to join cluster of
> [172.0.30.110:5001, 172.0.31.39:5001, 172.0.32.249:5001] 2017-05-30
> 09:19:06.996+0000 ERROR Failed to start Neo4j: Starting Neo4j failed:
> Component
> 'org.neo4j.server.database.LifecycleManagingDatabase@626a4cfa' was
> successfully initialized, but failed to start. Please see attached
> cause exception. Starting Neo4j failed: Component
> 'org.neo4j.server.database.LifecycleManagingDatabase@626a4cfa' was
> successfully initialized, but failed to start. Please see attached
> cause exception. org.neo4j.server.ServerStartupException: Starting
> Neo4j failed: Component
> 'org.neo4j.server.database.LifecycleManagingDatabase@626a4cfa' was
> successfully initialized, but failed to start. Please see attached
> cause exception.
>         at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)
>         at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:215)
>         at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:90)
>         at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:67)
>         at org.neo4j.server.enterprise.EnterpriseEntryPoint.main(EnterpriseEntryPoint.java:32)
> Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component
> 'org.neo4j.server.database.LifecycleManagingDatabase@626a4cfa' was
> successfully initialized, but failed to start. Please see attached
> cause exception.
>         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
>         at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
>         at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:187)
>         ... 3 more Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.ha.factory.HighlyAvailableFacadeFactory,
> /var/lib/neo4j/data/databases/fraud.db
>         at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:144)
>         at org.neo4j.kernel.ha.factory.HighlyAvailableFacadeFactory.newFacade(HighlyAvailableFacadeFactory.java:42)
>         at org.neo4j.kernel.ha.HighlyAvailableGraphDatabase.<init>(HighlyAvailableGraphDatabase.java:41)
>         at org.neo4j.server.enterprise.EnterpriseNeoServer.lambda$static$0(EnterpriseNeoServer.java:80)
>         at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:89)
>         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
>         ... 5 more Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component
> 'org.neo4j.cluster.client.ClusterJoin@5d2e6054' was successfully
> initialized, but failed to start. Please see attached cause exception.
>         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
>         at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
>         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
>         at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
>         at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:140)
>         ... 10 more Caused by: java.util.concurrent.TimeoutException: Conversation-response mapping:
> {3/13#=ResponseFuture{conversationId='3/13#',
> initiatedByMessageType=join, response=null}}
>         at org.neo4j.cluster.statemachine.StateMachineProxyFactory$ResponseFuture.get(StateMachineProxyFactory.java:314)
>         at org.neo4j.cluster.client.ClusterJoin.joinByConfig(ClusterJoin.java:143)
>         at org.neo4j.cluster.client.ClusterJoin.start(ClusterJoin.java:82)
>         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
>         ... 14 more ubuntu@ip-172-0-32-249:~$ sudo /usr/bin/docker rm neo4j neo4j ubuntu@ip-172-0-32-249:~$ sudo /usr/bin/docker run
> --publish=7474:7474 --publish=7687:7687 --publish=5001:5001 --volume=/var/lib/neo4j/data:/data --volume=/var/lib/neo4j/logs:/logs --volume=/var/lib/neo4j/conf:/conf --name=neo4j neo4j:3.0-enterprise Starting Neo4j. 2017-05-30 09:20:17.877+0000 INFO  No SSL certificate
> found, generating a self-signed certificate.. 2017-05-30
> 09:20:18.387+0000 INFO  Starting... 2017-05-30 09:20:18.676+0000 INFO 
> Write transactions to database disabled 2017-05-30 09:20:19.036+0000
> INFO  Bolt enabled on 0.0.0.0:7687. 2017-05-30 09:20:19.046+0000 INFO 
> Initiating metrics... 2017-05-30 09:20:19.779+0000 INFO  Attempting to
> join cluster of [172.0.30.110:5001, 172.0.31.39:5001,
> 172.0.32.249:5001] 2017-05-30 09:20:49.851+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component
> 'org.neo4j.server.database.LifecycleManagingDatabase@1b826a22' was
> successfully initialized, but failed to start. Please see attached
> cause exception. Starting Neo4j failed: Component
> 'org.neo4j.server.database.LifecycleManagingDatabase@1b826a22' was
> successfully initialized, but failed to start. Please see attached
> cause exception. org.neo4j.server.ServerStartupException: Starting
> Neo4j failed: Component
> 'org.neo4j.server.database.LifecycleManagingDatabase@1b826a22' was
> successfully initialized, but failed to start. Please see attached
> cause exception.
>         at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)
>         at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:215)
>         at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:90)
>         at org.neo4j.server.ServerBootstrapper.start(ServerBootstrapper.java:67)
>         at org.neo4j.server.enterprise.EnterpriseEntryPoint.main(EnterpriseEntryPoint.java:32)
> Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component
> 'org.neo4j.server.database.LifecycleManagingDatabase@1b826a22' was
> successfully initialized, but failed to start. Please see attached
> cause exception.
>         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
>         at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
>         at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:187)
>         ... 3 more Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.ha.factory.HighlyAvailableFacadeFactory,
> /var/lib/neo4j/data/databases/fraud.db
>         at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:144)
>         at org.neo4j.kernel.ha.factory.HighlyAvailableFacadeFactory.newFacade(HighlyAvailableFacadeFactory.java:42)
>         at org.neo4j.kernel.ha.HighlyAvailableGraphDatabase.<init>(HighlyAvailableGraphDatabase.java:41)
>         at org.neo4j.server.enterprise.EnterpriseNeoServer.lambda$static$0(EnterpriseNeoServer.java:80)
>         at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:89)
>         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
>         ... 5 more Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component
> 'org.neo4j.cluster.client.ClusterJoin@6c1c89e2' was successfully
> initialized, but failed to start. Please see attached cause exception.
>         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:444)
>         at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
>         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
>         at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:107)
>         at org.neo4j.kernel.impl.factory.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:140)
>         ... 10 more Caused by: java.util.concurrent.TimeoutException: Conversation-response mapping:
> {3/13#=ResponseFuture{conversationId='3/13#',
> initiatedByMessageType=join, response=null}}
>         at org.neo4j.cluster.statemachine.StateMachineProxyFactory$ResponseFuture.get(StateMachineProxyFactory.java:314)
>         at org.neo4j.cluster.client.ClusterJoin.joinByConfig(ClusterJoin.java:143)
>         at org.neo4j.cluster.client.ClusterJoin.start(ClusterJoin.java:82)
>         at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:434)
>         ... 14 more

Success (seen on one machine only):

2017-05-30 09:20:22.963+0000 INFO  Starting...
2017-05-30 09:20:23.319+0000 INFO  Write transactions to database disabled
2017-05-30 09:20:23.619+0000 INFO  Bolt enabled on 0.0.0.0:7687.
2017-05-30 09:20:23.628+0000 INFO  Initiating metrics...
2017-05-30 09:20:24.352+0000 INFO  Attempting to join cluster of [172.0.30.110:5001, 172.0.31.39:5001, 172.0.32.249:5001]
2017-05-30 09:20:31.382+0000 INFO  Could not join cluster of [172.0.30.110:5001,172.0.31.39:5001, 172.0.32.249:5001]
2017-05-30 09:20:31.382+0000 INFO  Creating new cluster with name [neo4j.ha]...
2017-05-30 09:20:31.396+0000 INFO  Instance 1 (this server)  entered the cluster
2017-05-30 09:20:31.401+0000 INFO  Instance 1 (this server)  was elected as coordinator
2017-05-30 09:20:31.413+0000 INFO  I am 1, moving to master
2017-05-30 09:20:31.465+0000 INFO  Instance 1 (this server)  was elected as coordinator
2017-05-30 09:20:31.492+0000 INFO  I am 1, successfully moved to master
2017-05-30 09:20:31.510+0000 INFO  Instance 1 (this server)  is available as master at ha://172.17.0.2:6001?serverId=1 with StoreId{creationTime=1496122128110,randomId=-2708986986476371425, storeVersion=15531981201765894, upgradeTime=1496122128110, upgradeId=1}
2017-05-30 09:20:31.612+0000 INFO  Instance 1 (this server)  is available as backup at backup://127.0.0.1:6362 with StoreId{creationTime=1496122128110, randomId=-2708986986476371425, storeVersion=15531981201765894, upgradeTime=1496122128110, upgradeId=1}
2017-05-30 09:20:31.709+0000 INFO  Database available for write transactions
2017-05-30 09:20:32.929+0000 INFO  Started.
2017-05-30 09:20:33.087+0000 INFO  Mounted REST API at: /db/manage
2017-05-30 09:20:33.828+0000 INFO  Remote interface available at http://0.0.0.0:7474/
neo4j
0 Answers
Related