【大数据环境准备】(五)kafka安装

本文涉及的产品
服务治理 MSE Sentinel/OpenSergo,Agent数量 不受限
简介: kafka安装

1、解压复制文件

[centos@hadoop10 module]$ mv kafka_2.12-3.0.0/ /data/module/kafka

[centos@hadoop10 kafka]$ vim ./config/server.properties
设置文件目录
[centos@hadoop10 kafka]$ mkdir datas

2、更改配置目录

路径 :data/module/kafka/config/server.properties

log.dirs=/data/module/kafka/datasck

zookeeper.connect=hadoop10:2181,hadoop11:2181,hadoop12:2181/kafka

num.partitions=1 //区分机器

3、设置环境变量

[centos@hadoop10 bin]$ vim /etc/profile.d/my_env.sh

KAFKA_HOME

export KAFKA_HOME=/data/module/kafka
export PATH=$PATH:$KAFKA_HOME/bin

4、分发配置

xsync kafka

5、批量启停文件

[atguigu@hadoop102 bin]$ vim kf.sh
脚本如下:
#! /bin/bash

case $1 in
"start"){
    for i in hadoop10 hadoop11 hadoop12
    do
        echo " --------启动 $i Kafka-------"
        ssh $i "/data/module/kafka/bin/kafka-server-start.sh -daemon /data/module/kafka/config/server.properties"
    done
};;
"stop"){
    for i in hadoop10 hadoop11 hadoop12
    do
        echo " --------停止 $i Kafka-------"
        ssh $i "/data/module/kafka/bin/kafka-server-stop.sh "
    done
};;
esac

6、问题

可能出现的第一种错误发生以下错误:

[2023-08-19 08:21:00,955] INFO [ReplicaManager broker=0] Shut down completely (kafka.server.ReplicaManager)
[2023-08-19 08:21:00,955] INFO [BrokerToControllerChannelManager broker=0 name=alterIsr]: Shutting down (kafka.server.BrokerToControllerRequestThread)
[2023-08-19 08:21:00,955] INFO [BrokerToControllerChannelManager broker=0 name=alterIsr]: Stopped (kafka.server.BrokerToControllerRequestThread)
[2023-08-19 08:21:00,956] INFO [BrokerToControllerChannelManager broker=0 name=alterIsr]: Shutdown completed (kafka.server.BrokerToControllerRequestThread)
[2023-08-19 08:21:00,963] INFO Broker to controller channel manager for alterIsr shutdown (kafka.server.BrokerToControllerChannelManagerImpl)
[2023-08-19 08:21:00,964] INFO [BrokerToControllerChannelManager broker=0 name=forwarding]: Shutting down (kafka.server.BrokerToControllerRequestThread)
[2023-08-19 08:21:00,965] INFO [BrokerToControllerChannelManager broker=0 name=forwarding]: Stopped (kafka.server.BrokerToControllerRequestThread)
[2023-08-19 08:21:00,965] INFO [BrokerToControllerChannelManager broker=0 name=forwarding]: Shutdown completed (kafka.server.BrokerToControllerRequestThread)
[2023-08-19 08:21:00,966] INFO Broker to controller channel manager for forwarding shutdown (kafka.server.BrokerToControllerChannelManagerImpl)
[2023-08-19 08:21:00,966] INFO Shutting down. (kafka.log.LogManager)
[2023-08-19 08:21:00,993] INFO Shutdown complete. (kafka.log.LogManager)
[2023-08-19 08:21:00,995] INFO [feature-zk-node-event-process-thread]: Shutting down (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2023-08-19 08:21:01,008] INFO [feature-zk-node-event-process-thread]: Stopped (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2023-08-19 08:21:01,009] INFO [feature-zk-node-event-process-thread]: Shutdown completed (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2023-08-19 08:21:01,010] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2023-08-19 08:21:01,013] WARN An exception was thrown while closing send thread for session 0x1000069232f0004. (org.apache.zookeeper.ClientCnxn)
EndOfStreamException: Unable to read additional data from server sessionid 0x1000069232f0004, likely server has closed socket
    at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)
[2023-08-19 08:21:01,115] INFO Session: 0x1000069232f0004 closed (org.apache.zookeeper.ZooKeeper)
[2023-08-19 08:21:01,115] INFO EventThread shut down for session: 0x1000069232f0004 (org.apache.zookeeper.ClientCnxn)
[2023-08-19 08:21:01,115] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)

原因 kafka server.properties配置配做 broker.id=1 broker.id=2 清空/datas

第二种错误:

could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient

关闭防火墙 systemctl stop firewalld

相关文章
|
7天前
|
SQL 分布式计算 大数据
请问本地安装了大数据计算MaxCompute studio,如何验证联通性及基本DDL操作呢?
请问本地安装了大数据计算MaxCompute studio,如何验证联通性及基本DDL操作呢?
32 0
|
7天前
|
消息中间件 Java Kafka
Kafka【环境搭建 01】kafka_2.12-2.6.0 单机版安装+参数配置及说明+添加到service服务+开机启动配置+验证+chkconfig配置说明(一篇入门kafka)
【2月更文挑战第19天】Kafka【环境搭建 01】kafka_2.12-2.6.0 单机版安装+参数配置及说明+添加到service服务+开机启动配置+验证+chkconfig配置说明(一篇入门kafka)
67 1
|
2天前
|
大数据 Linux 虚拟化
大数据软件基础(3) —— 在VMware上安装Linux集群
大数据软件基础(3) —— 在VMware上安装Linux集群
7 0
|
7天前
|
分布式计算 DataWorks Java
DataWorks产品使用合集之阿里云DataWorks专有云环境下,上传MaxCompute的UDF(用户自定义函数)的JAR包的步骤如何解决
DataWorks作为一站式的数据开发与治理平台,提供了从数据采集、清洗、开发、调度、服务化、质量监控到安全管理的全套解决方案,帮助企业构建高效、规范、安全的大数据处理体系。以下是对DataWorks产品使用合集的概述,涵盖数据处理的各个环节。
24 0
|
7天前
|
大数据 Docker 容器
大数据 安装指南-----利用docker
大数据 安装指南-----利用docker
46 0
|
7天前
|
分布式计算 Hadoop 大数据
[大数据] mac 史上最简单 hadoop 安装过程
[大数据] mac 史上最简单 hadoop 安装过程
|
7天前
|
消息中间件 Kafka Linux
Kafka【付诸实践 03】Offset Explorer Kafka 的终极 UI 工具安装+简单上手+关键特性测试(一篇学会使用 Offset Explorer)
【2月更文挑战第21天】Kafka【付诸实践 03】Offset Explorer Kafka 的终极 UI 工具安装+简单上手+关键特性测试(一篇学会使用 Offset Explorer)
274 2
|
7天前
|
消息中间件 Kafka Apache
【Kafka专栏】windows搭建Kafka环境 & 详细教程(01)
【Kafka专栏】windows搭建Kafka环境 & 详细教程(01)
104 2
|
7天前
|
消息中间件 Java Kafka
docker安装kafka(wurstmeister)
docker安装kafka(wurstmeister)
177 0
|
7天前
|
缓存 Java 大数据
CDH大数据环境参数优化指南
CDH大数据环境参数优化指南

热门文章

最新文章