【大数据环境准备】(五)kafka安装

本文涉及的产品
服务治理 MSE Sentinel/OpenSergo,Agent数量 不受限
云原生大数据计算服务 MaxCompute,5000CU*H 100GB 3个月
注册配置 MSE Nacos/ZooKeeper,118元/月
简介: kafka安装

1、解压复制文件

[centos@hadoop10 module]$ mv kafka_2.12-3.0.0/ /data/module/kafka

[centos@hadoop10 kafka]$ vim ./config/server.properties
设置文件目录
[centos@hadoop10 kafka]$ mkdir datas

2、更改配置目录

路径 :data/module/kafka/config/server.properties

log.dirs=/data/module/kafka/datasck

zookeeper.connect=hadoop10:2181,hadoop11:2181,hadoop12:2181/kafka

num.partitions=1 //区分机器

3、设置环境变量

[centos@hadoop10 bin]$ vim /etc/profile.d/my_env.sh

KAFKA_HOME

export KAFKA_HOME=/data/module/kafka
export PATH=$PATH:$KAFKA_HOME/bin

4、分发配置

xsync kafka

5、批量启停文件

[atguigu@hadoop102 bin]$ vim kf.sh
脚本如下:
#! /bin/bash

case $1 in
"start"){
    for i in hadoop10 hadoop11 hadoop12
    do
        echo " --------启动 $i Kafka-------"
        ssh $i "/data/module/kafka/bin/kafka-server-start.sh -daemon /data/module/kafka/config/server.properties"
    done
};;
"stop"){
    for i in hadoop10 hadoop11 hadoop12
    do
        echo " --------停止 $i Kafka-------"
        ssh $i "/data/module/kafka/bin/kafka-server-stop.sh "
    done
};;
esac

6、问题

可能出现的第一种错误发生以下错误:

[2023-08-19 08:21:00,955] INFO [ReplicaManager broker=0] Shut down completely (kafka.server.ReplicaManager)
[2023-08-19 08:21:00,955] INFO [BrokerToControllerChannelManager broker=0 name=alterIsr]: Shutting down (kafka.server.BrokerToControllerRequestThread)
[2023-08-19 08:21:00,955] INFO [BrokerToControllerChannelManager broker=0 name=alterIsr]: Stopped (kafka.server.BrokerToControllerRequestThread)
[2023-08-19 08:21:00,956] INFO [BrokerToControllerChannelManager broker=0 name=alterIsr]: Shutdown completed (kafka.server.BrokerToControllerRequestThread)
[2023-08-19 08:21:00,963] INFO Broker to controller channel manager for alterIsr shutdown (kafka.server.BrokerToControllerChannelManagerImpl)
[2023-08-19 08:21:00,964] INFO [BrokerToControllerChannelManager broker=0 name=forwarding]: Shutting down (kafka.server.BrokerToControllerRequestThread)
[2023-08-19 08:21:00,965] INFO [BrokerToControllerChannelManager broker=0 name=forwarding]: Stopped (kafka.server.BrokerToControllerRequestThread)
[2023-08-19 08:21:00,965] INFO [BrokerToControllerChannelManager broker=0 name=forwarding]: Shutdown completed (kafka.server.BrokerToControllerRequestThread)
[2023-08-19 08:21:00,966] INFO Broker to controller channel manager for forwarding shutdown (kafka.server.BrokerToControllerChannelManagerImpl)
[2023-08-19 08:21:00,966] INFO Shutting down. (kafka.log.LogManager)
[2023-08-19 08:21:00,993] INFO Shutdown complete. (kafka.log.LogManager)
[2023-08-19 08:21:00,995] INFO [feature-zk-node-event-process-thread]: Shutting down (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2023-08-19 08:21:01,008] INFO [feature-zk-node-event-process-thread]: Stopped (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2023-08-19 08:21:01,009] INFO [feature-zk-node-event-process-thread]: Shutdown completed (kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
[2023-08-19 08:21:01,010] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2023-08-19 08:21:01,013] WARN An exception was thrown while closing send thread for session 0x1000069232f0004. (org.apache.zookeeper.ClientCnxn)
EndOfStreamException: Unable to read additional data from server sessionid 0x1000069232f0004, likely server has closed socket
    at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290)
[2023-08-19 08:21:01,115] INFO Session: 0x1000069232f0004 closed (org.apache.zookeeper.ZooKeeper)
[2023-08-19 08:21:01,115] INFO EventThread shut down for session: 0x1000069232f0004 (org.apache.zookeeper.ClientCnxn)
[2023-08-19 08:21:01,115] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)

原因 kafka server.properties配置配做 broker.id=1 broker.id=2 清空/datas

第二种错误:

could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient

关闭防火墙 systemctl stop firewalld

相关文章
|
2月前
|
消息中间件 关系型数据库 MySQL
大数据-117 - Flink DataStream Sink 案例:写出到MySQL、写出到Kafka
大数据-117 - Flink DataStream Sink 案例:写出到MySQL、写出到Kafka
171 0
|
2月前
|
消息中间件 分布式计算 NoSQL
大数据-104 Spark Streaming Kafka Offset Scala实现Redis管理Offset并更新
大数据-104 Spark Streaming Kafka Offset Scala实现Redis管理Offset并更新
43 0
|
28天前
|
存储 大数据 数据处理
大数据环境下的性能优化策略
大数据环境下的性能优化策略
43 2
|
2月前
|
分布式计算 Hadoop 大数据
大数据体系知识学习(一):PySpark和Hadoop环境的搭建与测试
这篇文章是关于大数据体系知识学习的,主要介绍了Apache Spark的基本概念、特点、组件,以及如何安装配置Java、PySpark和Hadoop环境。文章还提供了详细的安装步骤和测试代码,帮助读者搭建和测试大数据环境。
66 1
|
2月前
|
消息中间件 存储 druid
大数据-156 Apache Druid 案例实战 Scala Kafka 订单统计
大数据-156 Apache Druid 案例实战 Scala Kafka 订单统计
44 3
|
2月前
|
消息中间件 druid 大数据
大数据-153 Apache Druid 案例 从 Kafka 中加载数据并分析(二)
大数据-153 Apache Druid 案例 从 Kafka 中加载数据并分析(二)
34 2
|
2月前
|
消息中间件 分布式计算 druid
大数据-153 Apache Druid 案例 从 Kafka 中加载数据并分析(一)
大数据-153 Apache Druid 案例 从 Kafka 中加载数据并分析(一)
56 1
|
1月前
|
消息中间件 Ubuntu Java
Ubuntu系统上安装Apache Kafka
Ubuntu系统上安装Apache Kafka
|
2月前
|
消息中间件 NoSQL Kafka
大数据-116 - Flink DataStream Sink 原理、概念、常见Sink类型 配置与使用 附带案例1:消费Kafka写到Redis
大数据-116 - Flink DataStream Sink 原理、概念、常见Sink类型 配置与使用 附带案例1:消费Kafka写到Redis
176 0
|
2月前
|
消息中间件 资源调度 大数据
大数据-112 Flink DataStreamAPI 程序输入源 DataSource 基于文件、集合、Kafka连接器
大数据-112 Flink DataStreamAPI 程序输入源 DataSource 基于文件、集合、Kafka连接器
45 0