DolphinScheduler2.x 伪分布式部署

本文涉及的产品
RDS MySQL Serverless 基础系列,0.5-2RCU 50GB
云数据库 RDS MySQL,集群系列 2核4GB
推荐场景:
搭建个人博客
云数据库 RDS PostgreSQL,集群系列 2核4GB
简介: DolphinScheduler2.x 伪分布式部署

DolphinScheduler2.x 伪分布式部署

DolphinScheduler 部署说明

1 软硬件环境要求

1.1 操作系统版本要求

操作系统 版本
Red Hat Enterprise Linux 7.0 及以上
CentOS 7.0 及以上
Oracle Enterprise Linux 7.0 及以上
Ubuntu LTS 16.04 及以上

1.2 服务器硬件要求

CPU 内存 硬盘类型 网络 实例数量
4 核+ 8 GB+ SAS 千兆网卡 1+

2 部署模式

DolphinScheduler 支持多种部署模式,包括单机模式(Standalone)、伪集群模式(Pseudo- Cluster)、集群模式(Cluster)等。

2.1 单机模式

单机模式(standalone)模式下,所有服务均集中于一个 StandaloneServer 进程中,并且

其中内置了注册中心 Zookeeper 和数据库 H2 。只需配置 JDK 环境, 就可一键启动 DolphinScheduler,快速体验其功能。

2.2 伪集群模式

伪集群模式(Pseudo-Cluster)是在单台机器部署 DolphinScheduler 各项服务,该模式 下 master、worker、api server 、logger server 等服务都只在同一台机器上。Zookeeper 和数据 库需单独安装并进行相应配置。

2.3 集群模式

集群模式(Cluster)与伪集群模式的区别就是在多台机器部署 DolphinScheduler 各项服 务,并且 Master、Worker 等服务可配置多个。

第 3 章 DolphinScheduler 集群模式部署

3.1 集群规划

集群模式下,可配置多个 Master 及多个 Worker 。通常可配置 2~3 个 Master ,若干个 Worker 。由于集群资源有限,此处配置一个 Master,一个 Worker ,集群规划如下。

node1 master、worker

3.2 前置准备工作

(1)节点需部署 JDK(1.8+),并配置相关环境变量。

(2)需部署数据库,支持 MySQL(5.7+)或者 PostgreSQL(8.2.15+)。

(3)需部署 Zookeeper(3.4.6+)。

(4)节点需安装进程树分析工具psmisc。

sudo yum install -y psmisc

3.3 解压 DolphinScheduler 安装包

(1)上传 DolphinScheduler 安装包到 node1 节点的/opt/software 目录

(2)解压安装包到当前目录

注:解压目录并非最终的安装目录

tar -zxvf apache-dolphinscheduler-2.0.5-bin.tar.gz 

3.4 创建元数据库及用户

DolphinScheduler 元数据存储在关系型数据库中,故需创建相应的数据库和用户。

(1)创建数据库

(2)创建用户

*注:*

若出现以下错误信息,表明新建用户的密码过于简单。

ERROR 1819 (HY000): Your password does not satisfy the current policy requirements

可提高密码复杂度或者执行以下命令降低 MySQL 密码强度级别。

(3)赋予用户相应权限

mysql>    GRANT    ALL    PRIVILEGES    ON    dolphinscheduler.*    TO
'dolphinscheduler'@'%';  mysql> flush privileges;

具体命令如下:

CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 
DEFAULT COLLATE utf8_general_ci;
CREATE USER 'dolphinscheduler'@'%' IDENTIFIED BY 
'Mr^HYe]9cR]D';
GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 
'dolphinscheduler'@'%';
FLUSH PRIVILEGES;

3.5 配置一键部署脚本

修改解压目录下的 conf/config 目录下的 install_config.conf 文件。

vim conf/config/install_config.conf

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# ---------------------------------------------------------
# INSTALL MACHINE
# ---------------------------------------------------------
# A comma separated list of machine hostname or IP would be installed DolphinScheduler,
# including master, worker, api, alert. If you want to deploy in pseudo-distributed
# mode, just write a pseudo-distributed hostname
# Example for hostnames: ips="ds1,ds2,ds3,ds4,ds5", Example for IPs: ips="192.168.8.1,192.168.8.2,192.168.8.3,192.168.8.4,192.168.8.5"
ips="node1"
# Port of SSH protocol, default value is 22. For now we only support same port in all `ips` machine
# modify it if you use different ssh port
sshPort="22"
# A comma separated list of machine hostname or IP would be installed Master server, it
# must be a subset of configuration `ips`.
# Example for hostnames: masters="ds1,ds2", Example for IPs: masters="192.168.8.1,192.168.8.2"
masters="node1"
# A comma separated list of machine <hostname>:<workerGroup> or <IP>:<workerGroup>.All hostname or IP must be a
# subset of configuration `ips`, And workerGroup have default value as `default`, but we recommend you declare behind the hosts
# Example for hostnames: workers="ds1:default,ds2:default,ds3:default", Example for IPs: workers="192.168.8.1:default,192.168.8.2:default,192.168.8.3:default"
workers="node1:default"
# A comma separated list of machine hostname or IP would be installed Alert server, it
# must be a subset of configuration `ips`.
# Example for hostname: alertServer="ds3", Example for IP: alertServer="192.168.8.3"
alertServer="node1"
# A comma separated list of machine hostname or IP would be installed API server, it
# must be a subset of configuration `ips`.
# Example for hostname: apiServers="ds1", Example for IP: apiServers="192.168.8.1"
apiServers="node1"
# A comma separated list of machine hostname or IP would be installed Python gateway server, it
# must be a subset of configuration `ips`.
# Example for hostname: pythonGatewayServers="ds1", Example for IP: pythonGatewayServers="192.168.8.1"
#pythonGatewayServers="ds1"
# The directory to install DolphinScheduler for all machine we config above. It will automatically be created by `install.sh` script if not exists.
# Do not set this configuration same as the current path (pwd)
installPath="/opt/module/dolphinscheduler"
# The user to deploy DolphinScheduler for all machine we config above. For now user must create by yourself before running `install.sh`
# script. The user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled than the root directory needs
# to be created by this user
deployUser="root"
# The directory to store local data for all machine we config above. Make sure user `deployUser` have permissions to read and write this directory.
dataBasedirPath="/tmp/dolphinscheduler"
# ---------------------------------------------------------
# DolphinScheduler ENV
# ---------------------------------------------------------
# JAVA_HOME, we recommend use same JAVA_HOME in all machine you going to install DolphinScheduler
# and this configuration only support one parameter so far.
javaHome="/export/server/jdk1.8.0_241/"
# DolphinScheduler API service port, also this is your DolphinScheduler UI component's URL port, default value is 12345
apiServerPort="12345"
# ---------------------------------------------------------
# Database
# NOTICE: If database value has special characters, such as `.*[]^${}\+?|()@#&`, Please add prefix `\` for escaping.
# ---------------------------------------------------------
# The type for the metadata database
# Supported values: ``postgresql``, ``mysql`, `h2``.
DATABASE_TYPE="mysql"
# 数据库类型
# Spring datasource url, following <HOST>:<PORT>/<database>?<parameter> format, If you using mysql, you could use jdbc
# string jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8 as example
SPRING_DATASOURCE_URL="jdbc:mysql://node1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8"
# Spring datasource username
SPRING_DATASOURCE_USERNAME="dolphinscheduler"
# 数据库用户名
# Spring datasource password
#SPRING_DATASOURCE_PASSWORD=${SPRING_DATASOURCE_PASSWORD:-""}
SPRING_DATASOURCE_PASSWORD="Mr^HYe]9cR]D"
# 数据库密码
# ---------------------------------------------------------
# Registry Server
# ---------------------------------------------------------
# Registry Server plugin name, should be a substring of `registryPluginDir`, DolphinScheduler use this for verifying configuration consistency
registryPluginName="zookeeper"
# 注册中心地址,即 Zookeeper 集群的地址
# Registry Server address.
registryServers="192.168.88.100:2181"
# Registry Namespace
registryNamespace="dolphinscheduler"
# DS 在 Zookeeper 的结点名称
# ---------------------------------------------------------
# Worker Task Server
# ---------------------------------------------------------
# Worker Task Server plugin dir. DolphinScheduler will find and load the worker task plugin jar package from this dir.
taskPluginDir="lib/plugin/task"
# resource storage type: HDFS, S3, NONE
resourceStorageType="HDFS"
# 资源存储类型
# resource store on HDFS/S3 path, resource file will store to this hdfs path, self configuration, please make sure the directory exists on hdfs and has read write permissions. "/dolphinscheduler" is recommended
resourceUploadPath="/dolphinscheduler"
# if resourceStorageType is HDFS,defaultFS write namenode address,HA, you need to put core-site.xml and hdfs-site.xml in the conf directory.
# if S3,write S3 address,HA,for example :s3a://dolphinscheduler,
# Note,S3 be sure to create the root directory /dolphinscheduler
defaultFS="hdfs://node1:8020"
# 默认文件系统
# if resourceStorageType is S3, the following three configuration is required, otherwise please ignore
s3Endpoint="http://192.168.xx.xx:9010"
s3AccessKey="xxxxxxxxxx"
s3SecretKey="xxxxxxxxxx"
# resourcemanager port, the default value is 8088 if not specified
resourceManagerHttpAddressPort="8088"
# yarn RM http 访问端口
# if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single node, keep this value empty
yarnHaIps=
# Yarn RM 高可用 ip,若未启用 RM 高可用,则将该值置空
# if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single node, you only need to replace 'yarnIp1' to actual resourcemanager hostname
singleYarnIp="node1"
# who has permission to create directory under HDFS/S3 root path
# Note: if kerberos is enabled, please config hdfsRootUser=
hdfsRootUser="ds"
# kerberos config
# whether kerberos starts, if kerberos starts, following four items need to config, otherwise please ignore
kerberosStartUp="false"
# kdc krb5 config file path
krb5ConfPath="$installPath/conf/krb5.conf"
# keytab username,watch out the @ sign should followd by \\
keytabUserName="hdfs-mycluster\\@ESZ.COM"
# username keytab path
keytabPath="$installPath/conf/hdfs.headless.keytab"
# kerberos expire time, the unit is hour
kerberosExpireTime="2"
# use sudo or not
sudoEnable="true"
# worker tenant auto create
workerTenantAutoCreate="false"

3.6 初始化数据库

(1)拷贝 MySQL 驱动到 DolphinScheduler 的解压目录下的 lib 中,要求使用 MySQL JDBC Driver 8.0.16。

cp /opt/software/mysql-connector-java-8.0.16.jar lib/

(2)执行数据库初始化脚本

数据库初始化脚本位 于 DolphinScheduler 解 压 目 录 下 的 script 目 录 中 , 即 /opt/software/ds/apache-dolphinscheduler-2.0.5-bin/script/。

[atguigu@node1 apache-dolphinscheduler-2.0.5-bin]$ script/create- dolphinscheduler.sh

3.7 一键部署 DolphinScheduler

(1)启动 Zookeeper 集群

[atguigu@node1 apache-dolphinscheduler-2.0.5-bin]$ zk.sh start

(2)一键部署并启动 DolphinScheduler

[atguigu@node1 apache-dolphinscheduler-2.0.5-bin]$ ./install.sh

(3)查看 DolphinScheduler 进程

(4)访问 DolphinScheduler UI

DolphinScheduler UI 地址为http://node1:12345/dolphinscheduler

初始用户的用户名为:admin,密码为 dolphinscheduler123

3.8 DolphinScheduler 启停命令

DolphinScheduler 的启停脚本均位于其安装目录的bin 目录下。

1)一键启停所有服务

注意同Hadoop 的启停脚本进行区分。

2)启停 Master

3)启停 Worker

4)启停 Api

5)启停 Logger

6)启停 Alert

QA

mater worker 进程未启动

tail -100f logs/dolphinscheduler-worker-server-node1.itcast.cn.out

zookeeper未启动

调整 zoo.cfg 配置后重新启动,成功。

相关实践学习
基于MSE实现微服务的全链路灰度
通过本场景的实验操作,您将了解并实现在线业务的微服务全链路灰度能力。
目录
相关文章
|
分布式计算 关系型数据库 MySQL
DolphinScheduler安装部署
DolphinScheduler安装部署
1295 0
|
SQL Kubernetes 监控
在 k8s 环境中使用 mysql 部署 dolphinscheduler (非 helm 的方式)
在 k8s 环境中使用 mysql 部署 dolphinscheduler (非 helm 的方式)
1977 0
|
1月前
|
资源调度 分布式计算 Hadoop
搭建YARN集群
文章介绍了如何搭建YARN集群,包括启动HDFS集群、修改YARN配置文件、启动ResourceManager和NodeManager节点,并提供了用于管理Hadoop集群的自定义脚本。
29 2
|
3月前
|
存储 关系型数据库 Java
极速体验DolphinScheduler 3.2.1 Standalone 版[一]
极速体验DolphinScheduler 3.2.1 Standalone 版[一]
51 0
|
3月前
|
Kubernetes 数据库 容器
DolphinScheduler3.2.1 集群部署(Cluster)[三]
DolphinScheduler3.2.1 集群部署(Cluster)[三]
45 0
|
4月前
|
SQL 资源调度 分布式计算
Apache DolphinScheduler 【安装部署】
Apache DolphinScheduler 【安装部署】
|
5月前
|
分布式计算 Ubuntu 调度
如何本地搭建开源分布式任务调度系统DolphinScheduler并远程访问
如何本地搭建开源分布式任务调度系统DolphinScheduler并远程访问
237 0
|
5月前
|
Ubuntu 调度 数据安全/隐私保护
使用Docker部署开源分布式任务调度系统DolphinScheduler
使用Docker部署开源分布式任务调度系统DolphinScheduler
使用Docker部署开源分布式任务调度系统DolphinScheduler
|
5月前
|
数据可视化 Linux 调度
DolphinScheduler【部署 01】分布式可视化工作流任务调度工具DolphinScheduler部署使用实例分享(一篇入门学会使用DolphinScheduler)
DolphinScheduler【部署 01】分布式可视化工作流任务调度工具DolphinScheduler部署使用实例分享(一篇入门学会使用DolphinScheduler)
658 0
|
5月前
|
Ubuntu 调度 数据安全/隐私保护
Docker部署开源分布式任务调度平台DolphinScheduler并实现远程访问办公
Docker部署开源分布式任务调度平台DolphinScheduler并实现远程访问办公