Elasticsearch on Docker - Failed to create enrollment token when generating API key

user_13543225 0
Viewed 2820

Going through the Elasticsearch docs for setting up Elasticsearch/Kibana with Docker, but I'm getting several errors. I follow the steps exactly. I'm running this on an Ubuntu 20.04 EC2 instance. What am I doing wrong?

Here's what I did:

  1. docker pull docker.elastic.co/elasticsearch/elasticsearch:8.0.0
  2. docker pull docker.elastic.co/kibana/kibana:8.0.0
  3. docker network create elastic
  4. docker run --name es01 --net elastic -p 9200:9200 -it docker.elastic.co/elasticsearch/elasticsearch:8.0.0

After step 4, Elasticsearch says:

A password is generated for the elastic user and output to the terminal, plus enrollment tokens for enrolling Kibana and adding additional nodes to your cluster.

I get neither. Instead, I get these error logs:

{"@timestamp":"2022-02-24T22:28:24.318Z", "log.level":"ERROR", "message":"Failed to create enrollment token when generating API key", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[a178a1e9d98b][generic][T#4]","log.logger":"org.elasticsearch.xpack.security.enrollment.InternalEnrollmentTokenGenerator","elasticsearch.cluster.uuid":"mC4ceJ2nT7i9-QF6V537Zg","elasticsearch.node.id":"IthIKocaSACunHatZxePPw","elasticsearch.node.name":"a178a1e9d98b","elasticsearch.cluster.name":"docker-cluster","error.type":"org.elasticsearch.action.UnavailableShardsException","error.message":"[.security-7][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.security-7][0]] containing [index {[.security][fobYLX8BZdXU5J2_mb_p], source[{\"doc_type\":\"api_key\",\"creation_time\":1645741644265,\"expiration_time\":1645743444265,\"api_key_invalidated\":false,\"api_key_hash\":\"{PBKDF2}10000$PbPNTKm9i5HBuHO+W9snM/+0C1sf4OGjE3xC1m3xKew=$oQXD/UOSgR/hDNHz1IgNKoVOG4Zi0LkiPQW3IMPnRtA=\",\"role_descriptors\":{\"create_enrollment_token\":{\"cluster\":[\"cluster:admin/xpack/security/enroll/node\"],\"indices\":[],\"applications\":[],\"run_as\":[],\"metadata\":{},\"type\":\"role\"}},\"limited_by_role_descriptors\":{\"superuser\":{\"cluster\":[\"all\"],\"indices\":[{\"names\":[\"*\"],\"privileges\":[\"all\"],\"allow_restricted_indices\":false},{\"names\":[\"*\"],\"privileges\":[\"monitor\",\"read\",\"view_index_metadata\",\"read_cross_cluster\"],\"allow_restricted_indices\":true}],\"applications\":[{\"application\":\"*\",\"privileges\":[\"*\"],\"resources\":[\"*\"]}],\"run_as\":[\"*\"],\"metadata\":{\"_reserved\":true},\"type\":\"role\"}},\"name\":\"enrollment_token_API_key_fYbYLX8BZdXU5J2_mb_p\",\"version\":8000099,\"metadata_flattened\":null,\"creator\":{\"principal\":\"_xpack_security\",\"full_name\":null,\"email\":null,\"metadata\":{},\"realm\":\"__attach\",\"realm_type\":\"__attach\"}}]}] blocking until refresh]","error.stack_trace":"org.elasticsearch.action.UnavailableShardsException: [.security-7][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.security-7][0]] containing [index {[.security][fobYLX8BZdXU5J2_mb_p], source[{\"doc_type\":\"api_key\",\"creation_time\":1645741644265,\"expiration_time\":1645743444265,\"api_key_invalidated\":false,\"api_key_hash\":\"{PBKDF2}10000$PbPNTKm9i5HBuHO+W9snM/+0C1sf4OGjE3xC1m3xKew=$oQXD/UOSgR/hDNHz1IgNKoVOG4Zi0LkiPQW3IMPnRtA=\",\"role_descriptors\":{\"create_enrollment_token\":{\"cluster\":[\"cluster:admin/xpack/security/enroll/node\"],\"indices\":[],\"applications\":[],\"run_as\":[],\"metadata\":{},\"type\":\"role\"}},\"limited_by_role_descriptors\":{\"superuser\":{\"cluster\":[\"all\"],\"indices\":[{\"names\":[\"*\"],\"privileges\":[\"all\"],\"allow_restricted_indices\":false},{\"names\":[\"*\"],\"privileges\":[\"monitor\",\"read\",\"view_index_metadata\",\"read_cross_cluster\"],\"allow_restricted_indices\":true}],\"applications\":[{\"application\":\"*\",\"privileges\":[\"*\"],\"resources\":[\"*\"]}],\"run_as\":[\"*\"],\"metadata\":{\"_reserved\":true},\"type\":\"role\"}},\"name\":\"enrollment_token_API_key_fYbYLX8BZdXU5J2_mb_p\",\"version\":8000099,\"metadata_flattened\":null,\"creator\":{\"principal\":\"_xpack_security\",\"full_name\":null,\"email\":null,\"metadata\":{},\"realm\":\"__attach\",\"realm_type\":\"__attach\"}}]}] blocking until refresh]\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retryBecauseUnavailable(TransportReplicationAction.java:1076)\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:872)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onTimeout(TransportReplicationAction.java:1031)\n\tat org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345)\n\tat org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263)\n\tat org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:651)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"}
{"@timestamp":"2022-02-24T22:28:47.612Z", "log.level":"ERROR", "message":"error downloading geoip database [GeoLite2-ASN.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[a178a1e9d98b][generic][T#6]","log.logger":"org.elasticsearch.ingest.geoip.GeoIpDownloader","elasticsearch.cluster.uuid":"mC4ceJ2nT7i9-QF6V537Zg","elasticsearch.node.id":"IthIKocaSACunHatZxePPw","elasticsearch.node.name":"a178a1e9d98b","elasticsearch.cluster.name":"docker-cluster","error.type":"org.elasticsearch.action.UnavailableShardsException","error.message":"[.geoip_databases][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.geoip_databases][0]] containing [index {[.geoip_databases][GeoLite2-ASN.mmdb_0_1645741637264], source[n/a, actual length: [1mb], max length: 2kb]}]]","error.stack_trace":"org.elasticsearch.action.UnavailableShardsException: [.geoip_databases][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.geoip_databases][0]] containing [index {[.geoip_databases][GeoLite2-ASN.mmdb_0_1645741637264], source[n/a, actual length: [1mb], max length: 2kb]}]]\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retryBecauseUnavailable(TransportReplicationAction.java:1076)\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:872)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onTimeout(TransportReplicationAction.java:1031)\n\tat org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345)\n\tat org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263)\n\tat org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:651)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"}
{"@timestamp":"2022-02-24T22:28:54.310Z", "log.level":"ERROR", "message":"Failed to generate credentials for the elastic built-in superuser", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[a178a1e9d98b][generic][T#7]","log.logger":"org.elasticsearch.xpack.security.InitialNodeSecurityAutoConfiguration","elasticsearch.cluster.uuid":"mC4ceJ2nT7i9-QF6V537Zg","elasticsearch.node.id":"IthIKocaSACunHatZxePPw","elasticsearch.node.name":"a178a1e9d98b","elasticsearch.cluster.name":"docker-cluster","error.type":"org.elasticsearch.action.UnavailableShardsException","error.message":"[.security-7][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.security-7][0]] containing [index {[.security][reserved-user-elastic], source[{\"password\":\"ff1DWkSBw4Cju0b8U7PM\",\"enabled\":true,\"type\":\"reserved-user\"}]}] and a refresh]","error.stack_trace":"org.elasticsearch.action.UnavailableShardsException: [.security-7][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.security-7][0]] containing [index {[.security][reserved-user-elastic], source[{\"password\":\"ff1DWkSBw4Cju0b8U7PM\",\"enabled\":true,\"type\":\"reserved-user\"}]}] and a refresh]\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.retryBecauseUnavailable(TransportReplicationAction.java:1076)\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:872)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onTimeout(TransportReplicationAction.java:1031)\n\tat org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345)\n\tat org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263)\n\tat org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:651)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"}

They are long messages. Here's the error portion of the above:

"Failed to create enrollment token when generating API key"
"error downloading geoip database [GeoLite2-ASN.mmdb]"
"Failed to generate credentials for the elastic built-in superuser"
"error downloading geoip database [GeoLite2-City.mmdb]"
kibana elasticsearch docker amazon-ec2
2 Answers

I assume your problem is because of the network, since you got failed when attempting to downloading geoip database and you use docker to run it. https://www.elastic.co/blog/docker-networking

When running Elasticsearch, you will need to ensure it publishes to an IP address that is reachable from outside the container; this can be configured via the setting network.publish_host.

user_47061600

I did not have enough storage space.

A gentleman on the Elasticsearch Slack channel was kind enough to point out that this was the real culprit:
"error.type": "org.elasticsearch.action.UnavailableShardsException", "error.message": "[.security-7][0] primary shard is not active Timeout: [1m],

I looked at my available host system storage space and found there was only 17G available! Cleaning up my Trash bin fixed the issue. Works now. Hopefully this helps someone else!

user_135432250
Related