Can't connect to a docker container using its IP address

user_9864539 0
Viewed 1395

I am using docker-compose to start up some containers which form a cluster of Solr and Zookeeper nodes.

This is my compose file (taken from here):

version: '3.8'
services:
  solr1:
    image: solr:8.7.0
    container_name: solr1
    ports:
     - "8981:8983"
    environment:
      - ZK_HOST=zoo1:2181,zoo2:2181,zoo3:2181
    networks:
      - solr
    depends_on:
      - zoo1
      - zoo2
      - zoo3

  solr2:
    image: solr:8.7.0
    container_name: solr2
    ports:
     - "8982:8983"
    environment:
      - ZK_HOST=zoo1:2181,zoo2:2181,zoo3:2181
    networks:
      - solr
    depends_on:
      - zoo1
      - zoo2
      - zoo3

  solr3:
    image: solr:8.7.0
    container_name: solr3
    ports:
     - "8983:8983"
    environment:
      - ZK_HOST=zoo1:2181,zoo2:2181,zoo3:2181
    networks:
      - solr
    depends_on:
      - zoo1
      - zoo2
      - zoo3

  zoo1:
    image: zookeeper:3.6.2
    container_name: zoo1
    restart: always
    hostname: zoo1
    ports:
      - 2181:2181
      - 7001:7000
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181
      ZOO_4LW_COMMANDS_WHITELIST: mntr, conf, ruok
      ZOO_CFG_EXTRA: "metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider metricsProvider.httpPort=7000 metricsProvider.exportJvmInfo=true"
    networks:
      - solr

  zoo2:
    image: zookeeper:3.6.2
    container_name: zoo2
    restart: always
    hostname: zoo2
    ports:
      - 2182:2181
      - 7002:7000
    environment:
      ZOO_MY_ID: 2
      ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181
      ZOO_4LW_COMMANDS_WHITELIST: mntr, conf, ruok
      ZOO_CFG_EXTRA: "metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider metricsProvider.httpPort=7000 metricsProvider.exportJvmInfo=true"
    networks:
      - solr

  zoo3:
    image: zookeeper:3.6.2
    container_name: zoo3
    restart: always
    hostname: zoo3
    ports:
      - 2183:2181
      - 7003:7000
    environment:
      ZOO_MY_ID: 3
      ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181
      ZOO_4LW_COMMANDS_WHITELIST: mntr, conf, ruok
      ZOO_CFG_EXTRA: "metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider metricsProvider.httpPort=7000 metricsProvider.exportJvmInfo=true"
    networks:
      - solr

networks:
  solr:

If I run docker compose ls, I see that the project is correctly running:

NAME                STATUS
solr-cluster        running(6)

and this is the output of docker container ls:

CONTAINER ID   IMAGE          COMMAND                  CREATED        STATUS          PORTS                                                                                                                NAMES
d5333378e78c   25b74ae1e488   "docker-entrypoint.s…"   23 hours ago   Up 52 minutes   0.0.0.0:8983->8983/tcp, :::8983->8983/tcp                                                                            solr3
5fac27cd0443   25b74ae1e488   "docker-entrypoint.s…"   23 hours ago   Up 52 minutes   0.0.0.0:8982->8983/tcp, :::8982->8983/tcp                                                                            solr2
db37ef90ca98   25b74ae1e488   "docker-entrypoint.s…"   23 hours ago   Up 52 minutes   0.0.0.0:8981->8983/tcp, :::8981->8983/tcp                                                                            solr1
b8b55694e281   a72350516291   "/docker-entrypoint.…"   23 hours ago   Up 52 minutes   2888/tcp, 3888/tcp, 8080/tcp, 0.0.0.0:2183->2181/tcp, :::2183->2181/tcp, 0.0.0.0:7003->7000/tcp, :::7003->7000/tcp   zoo3
10885eafe4e8   a72350516291   "/docker-entrypoint.…"   23 hours ago   Up 52 minutes   2888/tcp, 3888/tcp, 8080/tcp, 0.0.0.0:2182->2181/tcp, :::2182->2181/tcp, 0.0.0.0:7002->7000/tcp, :::7002->7000/tcp   zoo2
558f8574c036   a72350516291   "/docker-entrypoint.…"   23 hours ago   Up 52 minutes   2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 8080/tcp, 0.0.0.0:7001->7000/tcp, :::7001->7000/tcp   zoo1

If I try to navigate a solr node with my browser using host localhost (e.g. http://localhost:8983/), the container responds correctly with the Solr Admin interface.

However, if I try to access with my browser the same container but from its IP (e.g. http://172.23.0.7:8983/, where 172.23.0.7 is the IP of solr3 container) I get a connection timeout. I also get a connection timeout if I try to ping that IP.

Why does this happen? Shouldn't both localhost and 172.23.0.7 hosts work?

I need to access my container from its IP because the Solr nodes registers to Zookeeper using their IP. So, since I can't access the containers from their IP, I get a ConnectionTimeout exception when I try to connect to them programmatically through the SolrJ APIs:

public static void main(String[] args)
{
    List<String> zkNodes = new ArrayList<>(3);

    zkNodes.add("localhost:2181");
    zkNodes.add("localhost:2182");
    zkNodes.add("localhost:2183");

    SolrClient solrClient = new CloudSolrClient.Builder(zkNodes, Optional.empty())
        .withConnectionTimeout(10000)
        .withSocketTimeout(10000)
        .build();

    try
    {
        CollectionAdminRequest.listCollections(solrClient);
    }
    catch (IOException | SolrServerException e)
    {
        e.printStackTrace();
    }
}

This is the exception I get:

org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: http://172.23.0.7:8983/solr
    at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:695)
    at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:266)
    at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
    at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:370)
    at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:298)
    at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1157)
    at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:918)
    at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:850)
    at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:214)
    at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:231)
    at org.apache.solr.client.solrj.request.CollectionAdminRequest.listCollections(CollectionAdminRequest.java:2690)
    at Main.main(Main.java:28)
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to 172.23.0.7:8983 [/172.23.0.7] failed: connect timed out
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to 172.23.0.7:8983 [/172.23.0.7] failed: connect timed out
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374)
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
    at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:571)
    ... 11 more
Caused by: java.net.SocketTimeoutException: connect timed out
Caused by: java.net.SocketTimeoutException: connect timed out
    at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
    at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
    ... 21 more

I don't know if it that may help, but anyway I am on a Windows host, so I cannot use the --network host flag for Docker.

solrj docker-compose docker apache-zookeeper solr
3 Answers

The 172.x.x.x IPs are IPs only reachable within the Docker network, that is between the Solr and Zookeeper nodes.

When you connect to Zookeeper from outside, Zookeeper doesn't know there is an "outside", it only knows IP of Solr servers in its network and send them back to you. But these IPs are not reachable from "outside".

One workaround can be to connect to Solr servers directly instead of using Zookeeper (with URLs like localhost:8981 and HttpSolrClient).

Or, you need to tell Solr servers to "present" themselves with the outside IP to Zookeeper.

user_53891270

I think the advertised address for solar instances are the IP's of docker instances and the ports are the default ones instead of the redirected ones. You should register the docker ports instead of the solr ports in the pods.

Maybe the solrcloud part of the solr.xml can help, but not sure...

https://solr.apache.org/guide/8_8/format-of-solr-xml.html

user_110093510

Even though Gaël J's answer solves the problem in the general case, I'll post an answer to explain what needs to be done for Solr.

First, you need to tell Solr what is its hostname, by using the SOLR_HOST environment variable. This is the hostname which Solr will use when it registers itself to Zookeeper and it's also the hostname that Zookeeper will send to its clients when they ask for a Solr URL.

In my case, such clients were:

  1. a SolrJ application which run on the Docker host machine (but outside from Docker);
  2. the other Solr nodes, which may ask to Zookeeper the URL of other nodes

To make the SOLR_HOST understandable by both of these clients, I used the value of services.solrX.container_name as the SOLR_HOST, and I have added that value also to my hosts file on the Docker host machine.

So, this is the docker-compose.yml file:

version: '3.7'
services:
  solr1:
    image: solr:8.7.0
    container_name: solr1
    ports:
     - "8981:8981"
    environment:
      - ZK_HOST=zoo1:2181
      - SOLR_HOST=solr1
      - SOLR_PORT=8981
    networks:
      - solr
    depends_on:
      - zoo1

  solr2:
    image: solr:8.7.0
    container_name: solr2
    ports:
     - "8982:8982"
    environment:
      - ZK_HOST=zoo1:2181
      - SOLR_HOST=solr2
      - SOLR_PORT=8982
    networks:
      - solr
    depends_on:
      - zoo1
      
  zoo1:
    image: zookeeper:3.6.2
    container_name: zoo1
    restart: unless-stopped
    hostname: zoo1
    ports:
      - 2181:2181
      - 7001:7000
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=zoo1:2888:3888;2181
      ZOO_4LW_COMMANDS_WHITELIST: mntr, conf, ruok
      ZOO_CFG_EXTRA: "metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider metricsProvider.httpPort=7000 metricsProvider.exportJvmInfo=true"
    networks:
      - solr

networks:
  solr:

and this is the hosts file:

# local solr cloud cluster
127.0.0.1       solr1
127.0.0.1       solr2

Note also that both solr1 and solr2 point to localhost, so they should have different ports (this is why the SOLR_PORT environment variable is needed; otherwise both nodes would use the default 8983 port).

user_98645390
Related