docker compose kafka

最后更新时间：2025年02月24日

docker kafka 配置文件

version: '3.8'
name: "kafka-service"  # https://docs.docker.com/compose/how-tos/project-name/

services:
  kafka-server0:
    image: bitnami/kafka:latest
    hostname: kafka0
    container_name: kafka0
    ports:
      - "9092:9094"
    environment:
      KAFKA_CFG_NODE_ID: 0
      KAFKA_CFG_BROKER_ID: 0

      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://0.0.0.0:9094
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://kafka0:9092,EXTERNAL://192.168.60.134:9092
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@kafka1:9093,2@kafka2:9093,0@kafka0:9093
      KAFKA_CFG_PROCESS_ROLES: broker,controller
#      KAFKA_CFG_LOG_DIRS: /var/lib/kafka/data
      KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR: 2
      KAFKA_CFG_MIN_INSYNC_REPLICAS: 1
      KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: true
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_KRAFT_CLUSTER_ID: d6c6ea9d1bfe423ebdc74579b10c61f0
    networks:
      kafka:
        ipv4_address: 172.16.0.2
    volumes:
      - ~/kafka-data/0:/bitnami/kafka

  kafka-server1:
    image: bitnami/kafka:latest
    hostname: kafka1
    container_name: kafka1
    ports:
      - "9094:9094"
    environment:
      KAFKA_CFG_NODE_ID: 1
      KAFKA_CFG_BROKER_ID: 1

      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://0.0.0.0:9094
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092,EXTERNAL://192.168.60.134:9094
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@kafka1:9093,2@kafka2:9093,0@kafka0:9093
      KAFKA_CFG_PROCESS_ROLES: broker,controller
      #      KAFKA_CFG_LOG_DIRS: /var/lib/kafka/data
      KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR: 2
      KAFKA_CFG_MIN_INSYNC_REPLICAS: 1
      KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: true
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_KRAFT_CLUSTER_ID: d6c6ea9d1bfe423ebdc74579b10c61f0
    networks:
      kafka:
        ipv4_address: 172.16.0.3
    volumes:
      - ~/kafka-data/1:/bitnami/kafka

  kafka-server2:
    image: bitnami/kafka:latest
    hostname: kafka2
    container_name: kafka2
    ports:
      - "9096:9094"
    environment:
      KAFKA_CFG_NODE_ID: 2
      KAFKA_CFG_BROKER_ID: 2

      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://0.0.0.0:9094
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9092,EXTERNAL://192.168.60.134:9096
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@kafka1:9093,2@kafka2:9093,0@kafka0:9093
      KAFKA_CFG_PROCESS_ROLES: broker,controller
      #      KAFKA_CFG_LOG_DIRS: /var/lib/kafka/data
      KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
      KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
      KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR: 2
      KAFKA_CFG_MIN_INSYNC_REPLICAS: 1
      KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: true
      KAFKA_CFG_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_KRAFT_CLUSTER_ID: d6c6ea9d1bfe423ebdc74579b10c61f0
    networks:
      kafka:
        ipv4_address: 172.16.0.4
    volumes:
      - ~/kafka-data/2:/bitnami/kafka

networks:
  kafka:
    driver: bridge
    name: kafka-net
    ipam:
      config:
        - subnet: 172.16.0.0/24
          gateway: 172.16.0.1

KAFKA_CFG_LISTENERS：

  • 配置的监听器将被用于监听网络请求，listener_name://host_name:port,listener_name2://host_name2:port2。
  • 可以同时配置多个，并且用逗号隔开，监听器的名称和端口必须是唯一的，host_name如果为空，则会绑定到默认的接口(网卡)。

KAFKA_CFG_ADVERTISED_LISTENERS：

  • ADVERTISED_LISTENERS 配置 Kafka broker 向客户端（生产者和消费者）报告的地址。
  • 这是客户端用来连接到 broker 的地址。
  • 例如：ADVERTISED_LISTENERS=PLAINTEXT://broker-hostname:9092 表示 broker 将向客户端报告 broker-hostname:9092 作为连接地址。

KAFKA_CFG_PROCESS_ROLES：

  • 如果PROCESS_ROLES = Broker，服务器在KRaft模式中充当 Broker。
  • 如果PROCESS_ROLES = Controller，服务器在KRaft模式下充当 Controller。
  • 如果PROCESS_ROLES = Broker,Controller，服务器在KRaft模式中同时充当 Broker 和Controller。
  • 如果PROCESS_ROLES没有设置，那么集群就假定是运行在ZooKeeper模式下。

KAFKA_CFG_CONTROLLER_QUORUM_VOTERS：

  • 标识有哪些节点是 Quorum 的投票者节点。所有想成为控制器的节点都需要包含在这个配置里面。
  • 需要包含每个节点的id。如KAFKA_CFG_CONTROLLER_QUORUM_VOTERS: 1@kafka1:9093,2@kafka2:9093,0@kafka0:9093

KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR：

  • 用于指定内部主题 __consumer_offsets 的副本因子。__consumer_offsets 主题用于存储消费者组的偏移量信息。
  • 设置 KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR: 3 表示 __consumer_offsets 主题将有 3 个副本。这意味着即使有两个 broker 发生故障，仍然可以从剩余的一个副本中读取消费者偏移量信息。

KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR：

  • 指定内部主题 __transaction_state 的副本因子。__transaction_state 主题用于存储事务状态信息。

KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR：

KAFKA_CFG_MIN_INSYNC_REPLICAS：

  • 指定在写操作（例如生产消息）被认为成功之前，必须确认写入的最小同步副本（ISR）数量。

问题：

  • 在外部网络能够正常使用KafkaProducer向kafka docker集群生产消息，但KafkaConsumer不能正常使用group_id消费消息（而不指定group_id，却能消费消息）。问题无法复现，在解决NotLeaderForPartitionError问题后，不在出现。

  • 挂载卷问题（待解决）：

    kafka0  | mkdir: cannot create directory '/bitnami/kafka/config': Permission denied
kafka2  | mkdir: cannot create directory '/bitnami/kafka/config': Permission denied
kafka1  | mkdir: cannot create directory '/bitnami/kafka/config': Permission denied
    NOTE: As this is a non-root container, the mounted files and directories must have the proper permissions for the UID 1001.

    sudo chmod 776 -R kafka-data/：为其他用户组给出可读可写权限。

  • 生产者 NotLeaderForPartitionError：KAFKA_CFG_ADVERTISED_LISTENERS配置错误：

    • 错误配置：

      kafka-server1:
    image: bitnami/kafka:latest
    hostname: kafka1
    container_name: kafka1
    ports:
      - "9092:9094"
    environment:
      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092,EXTERNAL://localhost:9094

    • 正确配置：

      kafka-server1:
    image: bitnami/kafka:latest
    hostname: kafka1
    container_name: kafka1
    ports:
      - "9092:9094"
    environment:
      KAFKA_CFG_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://0.0.0.0:9094
      KAFKA_CFG_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092,EXTERNAL://192.168.60.134:9092

    • 原因：

      • Note: To connect from an external machine, change localhost above to your host's external IP/hostname and include EXTERNAL://0.0.0.0:9094 in KAFKA_CFG_LISTENERS to allow for remote connections.

      • KAFKA_CFG_ADVERTISED_LISTENERS:

        • PLAINTEXT://kafka1:9092,EXTERNAL://localhost:9094 修改为PLAINTEXT://kafka1:9092,EXTERNAL://192.168.60.134:9092

        • 192.168.60.134为docker宿主机IP，9092为docker宿主机映射端口

          (.venv) ronie@ronie:~/PycharmProjects/evilbox$ docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS          PORTS                                                   NAMES
f2509ff75469   bitnami/kafka:latest   "/opt/bitnami/script…"   6 seconds ago    Up 6 seconds    9092/tcp, 0.0.0.0:9092->9094/tcp, [::]:9092->9094/tcp   kafka0

        • 因为docker的网络拓扑是这样的

          graph TD
    subgraph Docker_Host
        direction TB
        br0[Bridge Network: kafka-net]
        kafka0[Kafka Broker 0]
        kafka1[Kafka Broker 1]
        kafka2[Kafka Broker 2]
        br0 --- kafka0
        br0 --- kafka1
        br0 --- kafka2
    end
    subgraph External_Network
        direction TB
        ext[External Client ]
    end
    ext --- br0

声明：Hello World|版权所有，违者必究|如未注明，均为原创|本网站采用BY-NC-SA协议进行授权

转载：转载请注明原文链接 - docker compose kafka

我的朋友，理论是灰色的，而生活之树是常青的！