• 周四. 12月 1st, 2022

5G编程聚合网

5G时代下一个聚合的编程学习网

热门标签

Docker compose to build Kafka cluster

[db:作者]

1月 6, 2022

Environmental preparation

kafka rely on zookeeper, So build kafka Configuration required zookeeper. The grid information is as follows :

zookeeper 192.168.56.101:2181
kafka1 192.168.56.101:9092
kafka2 192.168.56.101:9093
kafka3 192.168.56.101:9094

 

 

 

 

 

 

Begin to build

1、 install docker-compose

curl -L http://mirror.azure.cn/docker-toolbox/linux/compose/1.25.4/docker-compose-Linux-x86_64 -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

2、 establish docker-compose.yaml file


version: '3.3'
services:
zookeeper:
image: wurstmeister/zookeeper
container_name: zookeeper
ports:
- 2181:2181
volumes:
- ./data/zookeeper/data:/data
- ./data/zookeeper/datalog:/datalog
- ./data/zookeeper/logs:/logs
restart: always
kafka1:
image: wurstmeister/kafka
depends_on:
- zookeeper
container_name: kafka1
ports:
- 9092:9092
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 192.168.56.101:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.56.101:9092
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
 volumes:
- /data/kafka1/data:/data
- /data/kafka1/log:/datalog
restart: unless-stopped
kafka2:
image: wurstmeister/kafka
depends_on:
- zookeeper
container_name: kafka2
ports:
- 9093:9093
environment:
KAFKA_BROKER_ID: 2
KAFKA_ZOOKEEPER_CONNECT: 192.168.56.101:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.56.101:9093
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9093
 volumes:
- /data/kafka2/data:/data
- /data/kafka2/log:/datalog
restart: unless-stopped
kafka3:
image: wurstmeister/kafka
depends_on:
- zookeeper
container_name: kafka3
ports:
- 9094:9094
environment:
KAFKA_BROKER_ID: 3
KAFKA_ZOOKEEPER_CONNECT: 192.168.56.101:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.56.101:9094
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9094
 volumes:
- /data/kafka3/data:/data
- /data/kafka3/log:/datalog
restart: unless-stopped

View Code

Parameter description :

    KAFKA_ZOOKEEPER_CONNECT: zk Service address

    KAFKA_ADVERTISED_LISTENERS: kafka Service address

3、 start-up

docker-compose up -d

4、 View the launched container :docker-compose ps

The above result indicates that the startup is normal ~

 

test

1、 Log in to kafka1 In container

docker-compose exec kafka1 bash

 2、 Create a topic: The name is first,3 Zones ,2 Copies

./opt/kafka_2.13-2.7.0/bin/kafka-topics.sh --create --topic first --zookeeper 192.168.56.101:2181 --partitions 3 --replication-factor 2

Be careful : The number of copies cannot exceed brokers Count ( Partitions can be exceeded ), Otherwise, the creation will fail .

3、 see topic list

./opt/kafka_2.13-2.7.0/bin/kafka-topics.sh --list --zookeeper 192.168.56.101:2181 

4、 see topic by first Details of

./opt/kafka_2.13-2.7.0/bin/kafka-topics.sh --describe --topic first --zookeeper 192.168.56.101:2181

5、 Create a producer , towards topic Send message in

./opt/kafka_2.13-2.7.0/bin/kafka-console-producer.sh --topic first --broker-list 192.168.56.101:9092 

6、 Log in to kafka2 perhaps kafka3 In container ( Refer to the first 1 Step ), Then create a consumer , receive topic The messages in the

./opt/kafka_2.13-2.7.0/bin/kafka-console-consumer.sh --topic first --bootstrap-server 192.168.56.101:9092 --from-beginning

Be careful :–from-beginning Read the message from the beginning , Without this parameter, the maximum offset read ( Read from the latest message )

 

summary

1、 Connection information

producer     –>     broker-list
kafka colony    –>     zookeeper
consumer   –>     bootstrap-server or zookeeper

0.9 Version before ,consumer It’s Lian Xiang zookeeper,0.9 After the version , Default connection bootstrap-server

 

    

 

发表回复

您的电子邮箱地址不会被公开。 必填项已用*标注