org.apache.kafka.clients.NetworkClient Error connecting to node broker:29092 (id: -1 rack: null) java.net.UnknownHostException: broker: Name or service not known

[dash7ou]

control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available.
rest-proxy | [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.clients.admin.internals.AdminMetadataManager - [AdminClient clientId=adminclient-1] Metadata update failed
rest-proxy | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1606501070911, tries=1, nextAllowedTryMs=1606501071012) timed out at 1606501070912 after 1 attempt(s)
rest-proxy | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available.
ksqldb-server | [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.clients.admin.internals.AdminMetadataManager - [AdminClient clientId=adminclient-1] Metadata update failed
ksqldb-server | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1606501071531, tries=1, nextAllowedTryMs=1606501071632) timed out at 1606501071532 after 1 attempt(s)
ksqldb-server | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.
schema-registry | [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.clients.admin.internals.AdminMetadataManager - [AdminClient clientId=adminclient-1] Metadata update failed
schema-registry | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1606501072795, tries=1, nextAllowedTryMs=1606501072897) timed out at 1606501072797 after 1 attempt(s)
schema-registry | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting to send the call.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Error connecting to node broker:29092 (id: -1 rack: null)
control-center | java.net.UnknownHostException: broker: Name or service not known
control-center | at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
control-center | at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929)
control-center | at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1515)
control-center | at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848)
control-center | at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
control-center | at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
control-center | at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
control-center | at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:958)
control-center | at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:294)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1039)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1281)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1224)
control-center | at java.base/java.lang.Thread.run(Thread.java:834)
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Error connecting to node broker:29092 (id: -1 rack: null)
control-center | java.net.UnknownHostException: broker
control-center | at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
control-center | at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
control-center | at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
control-center | at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
control-center | at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:958)
control-center | at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:294)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1039)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1281)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1224)
control-center | at java.base/java.lang.Thread.run(Thread.java:834)
control-center | [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.clients.admin.internals.AdminMetadataManager - [AdminClient clientId=adminclient-1] Metadata update failed
control-center | org.apache.kafka.common.errors.TimeoutException: Call(callName=fetchMetadata, deadlineMs=1606501079587, tries=1, nextAllowedTryMs=1606501079698) timed out at 1606501079598 after 1 attempt(s)
control-center | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Error connecting to node broker:29092 (id: -1 rack: null)
control-center | java.net.UnknownHostException: broker
control-center | at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
control-center | at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
control-center | at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
control-center | at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
control-center | at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:958)
control-center | at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:294)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1039)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1281)
control-center | at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1224)
control-center | at java.base/java.lang.Thread.run(Thread.java:834)
rest-proxy | [main] ERROR io.confluent.admin.utils.ClusterStatus - Error while getting broker list.
rest-proxy | java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1606501080901, tries=1, nextAllowedTryMs=1606501081005) timed out at 1606501080905 after 1 attempt(s)
rest-proxy | at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
rest-proxy | at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
rest-proxy | at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
rest-proxy | at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
rest-proxy | at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:149)
rest-proxy | at io.confluent.admin.utils.cli.KafkaReadyCommand.main(KafkaReadyCommand.java:150)
rest-proxy | Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1606501080901, tries=1, nextAllowedTryMs=1606501081005) timed out at 1606501080905 after 1 attempt(s)
rest-proxy | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
control-center | [kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Error connecting to node broker:29092 (id: -1 rack: null)
control-center | java.net.UnknownHostException: broker
control-center | at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797)
control-center | at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364)
control-center | at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298)
control-center | at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
control-center | at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)

[dash7ou]

@ybyzek I have an issue when running docker-compose file for cp-all-in-one folder, on ec2 amazon, it runs fine for me locally.

[cosbor11]

This happens for me too

[konfortes]

Same here for local run.
osx 10.15.5
Docker 3.0.3(51017)

[wscardua]

me too

osx 10.15.7
docker 3.0.4 (51218)

[ybyzek]

Things to check:

1. Is the broker up? (docker-compose ps)
2. Can the other container ping the broker container? (docker-compose exec control-center ping broker)
3. Are there any errors in the logs (search through docker-compose logs)
4. Which branch of this repo is checked out? Make sure to run latest (latest or 6.0.1-post)

[ybyzek]

@mhbrk from the error logs:

broker | java.lang.IllegalArgumentException: requirement failed: Each listener must have a different port, listeners: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:9092

It looks like 9092 is reused twice, this is not allowed as described in the message. So it appears your local docker-compose.yml file has been modified. Please update it to match what is in GitHub (which is validated to work) and try again.

[DavoBR]

about the port 29092, i am facing with the following error

container console output:

confluentinc_cp-server_6.1.0.txt

[OneCricketeer]

also it doesn't have - "29092:29092" in this part

broker hostname isn't resolvable by the host, anyway, so a port forward is not the correct location to debug the problem

As the logs say, your listeners config cannot use the same port. Without this, your broker dies as well as all other services, such as control center, that depend on it

You may explicitly define KAFKA_LISTENERS, otherwise the default behavior in the Docker image is to use the advertised listeners, and replace the host names with 0.0.0.0

These get printed in the logs when the container starts https://github.com/confluentinc/confluent-docker-utils/blob/master/confluent/docker_utils/cub.py#L469

[tooptoop4]

@dash7ou did u solve?

[nicolepastrana]

This is what worked for me in my M1 Chip:

• zookeeper image: confluentinc/cp-zookeeper:latest.arm64
• broker image: confluentinc/cp-server:latest.arm64
• schema-registry image: confluentinc/cp-schema-registry:latest.arm64
• connect image: cnfldemos/cp-server-connect-datagen:0.5.0-6.2.0 [I haven't tested with the latest cnfldemos/cp-server-connect-datagen:0.5.3-7.1.0]
• control-center image: confluentinc/cp-enterprise-control-center:latest.arm64
• ksqldb-server image: confluentinc/cp-ksqldb-server:latest.arm64
• ksqldb-cli image: confluentinc/cp-ksqldb-cli:latest.arm64
• ksql-datagen image: confluentinc/ksqldb-examples:latest.arm64
• rest-proxy image: confluentinc/cp-kafka-rest:latest.arm64

https://github.com/confluentinc/cp-all-in-one/blob/7.1.1-post/cp-all-in-one/docker-compose.yml