Spark修炼之道(进阶篇)——Spark入门到精通:第十五节 Kafka 0.8.2.1 集群搭建-阿里云开发者社区

开发者社区> 大数据> 正文
登录阅读全文

Spark修炼之道(进阶篇)——Spark入门到精通:第十五节 Kafka 0.8.2.1 集群搭建

简介: 作者:周志湖 微信号:zhouzhihubeyond 本节为下一节Kafka与Spark Streaming做铺垫 主要内容 1.kafka 集群搭建 1. kafka 集群搭建 kafka 安装与配置 到下面的地址下载:Scala 2.10 - kafka_2.10-0.8.2.1.tgz http://kafka.apache.org/downloa

作者:周志湖
微信号:zhouzhihubeyond

本节为下一节Kafka与Spark Streaming做铺垫

主要内容

1.kafka 集群搭建

1. kafka 集群搭建

  1. kafka 安装与配置

    到下面的地址下载:Scala 2.10 - kafka_2.10-0.8.2.1.tgz
    http://kafka.apache.org/downloads.html
    下载完成后,使用命令

tar -zxvf  kafka_2.10-0.8.2.1.tgz 

解压,解压后的目录如下
这里写图片描述

进入config目录,将server.properties文件内容如下:

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0

############################# Socket Server Settings #############################

# The port the socket server listens on
port=9092

# Hostname the broker will bind to. If not set, the server will bind to all interfaces
host.name=sparkmaster

//中间省略,默认配置即可
############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=sparkmaster:2181,sparkslave01:2181,sparkslave02:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000

将整个安装文件进行跨机器拷贝:

root@sparkmaster:/hadoopLearning# scp -r kafka_2.10-0.8.2.1/ sparkslave01:/hadoopLearning/ 
root@sparkmaster:/hadoopLearning# scp -r kafka_2.10-0.8.2.1/ sparkslave02:/hadoopLearning/ 

将sparkslave01机器上的server.properties文件内容如下:


############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1

############################# Socket Server Settings #############################

# The port the socket server listens on
port=9092

# Hostname the broker will bind to. If not set, the server will bind to all interfaces
host.name=sparkslave01

//中间省略,默认配置即可

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=sparkmaster:2181,sparkslave01:2181,sparkslave02:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000

将sparkslave02机器上的server.properties文件内容如下:

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=2

############################# Socket Server Settings #############################

# The port the socket server listens on
port=9092

# Hostname the broker will bind to. If not set, the server will bind to all interfaces
host.name=sparkslave02


//中间省略,默认配置即可

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=sparkmaster:2181,sparkslave01:2181,sparkslave02:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
  1. 启动Kafka集群
root@sparkslave02:/hadoopLearning/kafka_2.10-0.8.2.1# bin/kafka-server-start.sh config/server.properties 
root@sparkslave01:/hadoopLearning/kafka_2.10-0.8.2.1# bin/kafka-server-start.sh config/server.properties 
root@sparkmaster:/hadoopLearning/kafka_2.10-0.8.2.1# bin/kafka-server-start.sh config/server.properties 

这里写图片描述

3 创建topic
在sparkmaster机器上执行下列命令创建一个topic

root@sparkmaster:/hadoopLearning/kafka_2.10-0.8.2.1# bin/kafka-topics.sh --create --topic kafkatopictest --replication-factor 3 --partitions 2 --zookeeper sparkmaster:2181
Created topic "kafkatopictest".

4 发送消息至kafka
在sparkslave01机器上执行下列命令并向kafka发送消息

root@sparkslave01:/hadoopLearning/kafka_2.10-0.8.2.1# bin/kafka-console-producer.sh --broker-list sparkslave01:9092 --sync --topic kafkatopictest
Hello Kafka, I will test Spark Streaming on you next lesson

这里写图片描述

5 接收kafka发送来的消息

在sparkslave02机器上执行下列命令并接收kafka发送消息

root@sparkslave02:/hadoopLearning/kafka_2.10-0.8.2.1# bin/kafka-console-consumer.sh --zookeeper sparkmaster:2181 --topic kafkatopictest --from-beginning
Hello Kafka, I will test Spark Streaming on you next lesson

这里写图片描述

至此Kafka 集群搭建与测试完毕

下一节当中,我们将演示kafka如何与Spark Streaimg结合起来使用

版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。

分享:
大数据
使用钉钉扫一扫加入圈子
+ 订阅

大数据计算实践乐园,近距离学习前沿技术

其他文章
最新文章
相关文章