点一下关注吧!!!非常感谢!!持续更新!!!
目前已经更新到了:
Hadoop(已更完)
HDFS(已更完)
MapReduce(已更完)
Hive(已更完)
Flume(已更完)
Sqoop(已更完)
Zookeeper(已更完)
HBase(已更完)
Redis (已更完)
Kafka(已更完)
Spark(已更完)
Flink(已更完)
ClickHouse(已更完)
Kudu(正在更新…)
章节内容
上节我们完成了如下的内容:
Kudu 的基本介绍
Kudu 的架构概述
安装准备
硬件准备
一台或者多台跑kudu-master。建议一个Master(无容错机制)、三个Master(允许一个节点出错)、五个Master(允许两个节点出错)
一台或者多台机器跑kudu-server。当需要使用副本,至少需要三个节点运行kudu-server服务。
操作系统
Ubuntu云服务器(我的环境是三台小云服务器)
内核和文件系统支持 Hole Punching 选项。
NTP 服务
xfs or ext4 formatterd drivers
存储要求
尽量使用固态存储,显著提供kudu性能
环境说明
如果你使用的是CDH,需要ClouderaManager 5.4.3 及以上的版本。
三台云服务器,h121启动Master,h122和123启动TabletServer。
Dockerfile
https://github.com/apache/kudu/blob/master/docker/quickstart.yml
官方的配置文件为:
- 3个Master
- 4个TabletServer
# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. version: "3" services: kudu-master-1: image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest} ports: - "7051:7051" - "8051:8051" command: ["master"] volumes: - kudu-master-1:/var/lib/kudu environment: - KUDU_MASTERS=kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251 # TODO: Use `host.docker.internal` instead of KUDU_QUICKSTART_IP when it # works on Linux (https://github.com/docker/for-linux/issues/264) - > MASTER_ARGS=--fs_wal_dir=/var/lib/kudu/master --rpc_bind_addresses=0.0.0.0:7051 --rpc_advertised_addresses=${KUDU_QUICKSTART_IP:?Please set KUDU_QUICKSTART_IP environment variable}:7051 --webserver_port=8051 --webserver_advertised_addresses=${KUDU_QUICKSTART_IP}:8051 --webserver_doc_root=/opt/kudu/www --stderrthreshold=0 --use_hybrid_clock=false --unlock_unsafe_flags=true kudu-master-2: image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest} ports: - "7151:7151" - "8151:8151" command: ["master"] volumes: - kudu-master-2:/var/lib/kudu environment: - KUDU_MASTERS=kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251 - > MASTER_ARGS=--fs_wal_dir=/var/lib/kudu/master --rpc_bind_addresses=0.0.0.0:7151 --rpc_advertised_addresses=${KUDU_QUICKSTART_IP}:7151 --webserver_port=8151 --webserver_advertised_addresses=${KUDU_QUICKSTART_IP}:8151 --webserver_doc_root=/opt/kudu/www --stderrthreshold=0 --use_hybrid_clock=false --unlock_unsafe_flags=true kudu-master-3: image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest} ports: - "7251:7251" - "8251:8251" command: ["master"] volumes: - kudu-master-3:/var/lib/kudu environment: - KUDU_MASTERS=kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251 - > MASTER_ARGS=--fs_wal_dir=/var/lib/kudu/master --rpc_bind_addresses=0.0.0.0:7251 --rpc_advertised_addresses=${KUDU_QUICKSTART_IP}:7251 --webserver_port=8251 --webserver_advertised_addresses=${KUDU_QUICKSTART_IP}:8251 --webserver_doc_root=/opt/kudu/www --stderrthreshold=0 --use_hybrid_clock=false --unlock_unsafe_flags=true kudu-tserver-1: image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest} depends_on: - kudu-master-1 - kudu-master-2 - kudu-master-3 ports: - "7050:7050" - "8050:8050" command: ["tserver"] volumes: - kudu-tserver-1:/var/lib/kudu environment: - KUDU_MASTERS=kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251 - > TSERVER_ARGS=--fs_wal_dir=/var/lib/kudu/tserver --rpc_bind_addresses=0.0.0.0:7050 --rpc_advertised_addresses=${KUDU_QUICKSTART_IP}:7050 --webserver_port=8050 --webserver_advertised_addresses=${KUDU_QUICKSTART_IP}:8050 --webserver_doc_root=/opt/kudu/www --stderrthreshold=0 --use_hybrid_clock=false --unlock_unsafe_flags=true kudu-tserver-2: image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest} depends_on: - kudu-master-1 - kudu-master-2 - kudu-master-3 ports: - "7150:7150" - "8150:8150" command: ["tserver"] volumes: - kudu-tserver-2:/var/lib/kudu environment: - KUDU_MASTERS=kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251 - > TSERVER_ARGS=--fs_wal_dir=/var/lib/kudu/tserver --rpc_bind_addresses=0.0.0.0:7150 --rpc_advertised_addresses=${KUDU_QUICKSTART_IP}:7150 --webserver_port=8150 --webserver_advertised_addresses=${KUDU_QUICKSTART_IP}:8150 --webserver_doc_root=/opt/kudu/www --stderrthreshold=0 --use_hybrid_clock=false --unlock_unsafe_flags=true kudu-tserver-3: image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest} depends_on: - kudu-master-1 - kudu-master-2 - kudu-master-3 ports: - "7250:7250" - "8250:8250" command: ["tserver"] volumes: - kudu-tserver-3:/var/lib/kudu environment: - KUDU_MASTERS=kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251 - > TSERVER_ARGS=--fs_wal_dir=/var/lib/kudu/tserver --rpc_bind_addresses=0.0.0.0:7250 --rpc_advertised_addresses=${KUDU_QUICKSTART_IP}:7250 --webserver_port=8250 --webserver_advertised_addresses=${KUDU_QUICKSTART_IP}:8250 --webserver_doc_root=/opt/kudu/www --stderrthreshold=0 --use_hybrid_clock=false --unlock_unsafe_flags=true kudu-tserver-4: image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest} depends_on: - kudu-master-1 - kudu-master-2 - kudu-master-3 ports: - "7350:7350" - "8350:8350" command: ["tserver"] volumes: - kudu-tserver-4:/var/lib/kudu environment: - KUDU_MASTERS=kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251 - > TSERVER_ARGS=--fs_wal_dir=/var/lib/kudu/tserver --rpc_bind_addresses=0.0.0.0:7350 --rpc_advertised_addresses=${KUDU_QUICKSTART_IP}:7350 --webserver_port=8350 --webserver_advertised_addresses=${KUDU_QUICKSTART_IP}:8350 --webserver_doc_root=/opt/kudu/www --stderrthreshold=0 --use_hybrid_clock=false --unlock_unsafe_flags=true kudu-tserver-5: image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest} depends_on: - kudu-master-1 - kudu-master-2 - kudu-master-3 ports: - "7450:7450" - "8450:8450" command: ["tserver"] volumes: - kudu-tserver-5:/var/lib/kudu environment: - KUDU_MASTERS=kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251 - > TSERVER_ARGS=--fs_wal_dir=/var/lib/kudu/tserver --rpc_bind_addresses=0.0.0.0:7450 --rpc_advertised_addresses=${KUDU_QUICKSTART_IP}:7450 --webserver_port=8450 --webserver_advertised_addresses=${KUDU_QUICKSTART_IP}:8450 --webserver_doc_root=/opt/kudu/www --stderrthreshold=0 --use_hybrid_clock=false --unlock_unsafe_flags=true volumes: kudu-master-1: kudu-master-2: kudu-master-3: kudu-tserver-1: kudu-tserver-2: kudu-tserver-3: kudu-tserver-4: kudu-tserver-5:
内容解释
version: “3”:指定使用 Docker Compose 文件的版本。版本 3 是当前常用的版本,支持多种高级功能,适用于 Docker 1.13.0 及更高版本。
services:定义了多个服务,每个服务对应一个 Docker 容器。在本配置中,定义了多个 Kudu Master 和 Kudu Tablet Server(简称 TServer)。
volumes:定义了持久化存储卷,用于存储各个服务的数据,确保即使容器重新创建或删除,数据仍然保留。
3 个 Kudu Master 节点:
kudu-master-1
kudu-master-2
kudu-master-3
5 个 Kudu Tablet Server 节点:
kudu-tserver-1
kudu-tserver-2
kudu-tserver-3
kudu-tserver-4
kudu-tserver-5
Master
每个 Master 节点的配置基本相同,只是端口和持久化卷不同。以下以 kudu-master-1 为例进行解释:
kudu-master-1: image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest} ports: - "7051:7051" - "8051:8051" command: ["master"] volumes: - kudu-master-1:/var/lib/kudu environment: - KUDU_MASTERS=kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251 - > MASTER_ARGS=--fs_wal_dir=/var/lib/kudu/master --rpc_bind_addresses=0.0.0.0:7051 --rpc_advertised_addresses=${KUDU_QUICKSTART_IP:?Please set KUDU_QUICKSTART_IP environment variable}:7051 --webserver_port=8051 --webserver_advertised_addresses=${KUDU_QUICKSTART_IP}:8051 --webserver_doc_root=/opt/kudu/www --stderrthreshold=0 --use_hybrid_clock=false --unlock_unsafe_flags=true
udu-master-1:服务名称,标识第一个 Kudu Master 节点。
image: apache/kudu:${KUDU_QUICKSTART_VERSION:-latest}:指定使用的 Docker 镜像。apache/kudu 是官方 Kudu 镜像。
${KUDU_QUICKSTART_VERSION:-latest}:使用环境变量 KUDU_QUICKSTART_VERSION 指定版本,如果未设置,则默认使用 latest。
ports:“7051:7051”:将主机的 7051 端口映射到容器的 7051 端口,用于 Kudu Master 的 RPC 通信。“8051:8051”:将主机的 8051 端口映射到容器的 8051 端口,用于 Kudu Master 的 Web 界面。
command: [“master”]:指定容器启动时运行的命令,这里启动 Kudu Master 进程。
volumes:kudu-master-1:/var/lib/kudu:将 Docker 卷 kudu-master-1 挂载到容器内的 /var/lib/kudu 目录,用于持久化存储数据。
KUDU_MASTERS:定义了集群中所有 Master 节点的地址,格式为 hostname:port,以逗号分隔。
启动服务
export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 | awk '{print $2}' | tail -1) docker-compose up -d
访问服务
http://localhost:8051/
这样我们的服务就顺利的启动了: