测试环境搭建整套大数据系统(五:搭建dolphinschduler3.1.9)

news/2024/7/27 22:18:41

一:官网下载安装包。

官网地址:

https://dolphinscheduler.apache.org/zh-cn/docs/3.1.9/architecture/configuration
在这里插入图片描述
在这里插入图片描述

二:进行解压。

root@hadoop100:/opt# tar zxvf dolphinscheduler-3.1.9-bin.tar.gz /opt

三:添加jar包。

1. 添加zk jar包。

进入 zookeeper 的安装目录,将 zoo_sample.cfg 配置文件复制到 conf/zoo.cfg,并将 conf/zoo.cfg 中 dataDir 中的值改成 dataDir=./tmp/zookeeper。
需要在alert-server,api-server,master-server,tools,worker-server这五个目录全部放进去。

2. 添加mysql8.0.6以上的jar包。

即使你使用的mysql是5.7,也要使用8.0.16以上的驱动。
将驱动包放到alert-server,api-server,master-server,tools,worker-server五个目录的libs目录下。
在这里插入图片描述

四:修改配置文件。

1. 修改dolphinscheduler_env.sh

cd /opt/apache-dolphinscheduler-3.1.9-bin/bin/env
vim dolphinscheduler_env.sh

mysql,zk,java地址必须修改。

# JAVA_HOME, will use it to start DolphinScheduler server
export JAVA_HOME=${JAVA_HOME:-/opt/jdk1.8.0_211}# Database related configuration, set database type, username and password
#export DATABASE=${DATABASE:-postgresql}
#export SPRING_PROFILES_ACTIVE=${DATABASE}
#export SPRING_DATASOURCE_URL
#export SPRING_DATASOURCE_USERNAME
#export SPRING_DATASOURCE_PASSWORD
export DATABASE=${DATABASE:-mysql}
export SPRING_PROFILES_ACTIVE=${DATABASE}
export SPRING_DATASOURCE_URL="jdbc:mysql://192.168.50.60:3306/dolphinscheduler?createDatabaseIfNotExist=true"
export SPRING_DATASOURCE_USERNAME="dolphinscheduler"
export SPRING_DATASOURCE_PASSWORD="dolphinscheduler"# DolphinScheduler server related configuration
export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none}
export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC}
export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10}# Registry center configuration, determines the type and link of the registry center
export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper}
export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-10.15.250.196:2181,10.15.250.202:2181,10.15.250.220:2181}export HADOOP_HOME=${HADOOP_HOME:-/opt/hadoop}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-/opt/hadoop/etc/hadoop}
export SPARK_HOME=${SPARK_HOME:-/opt/soft/spark}
export PYTHON_LAUNCHER=${PYTHON_LAUNCHER:-/opt/soft/python}
export HIVE_HOME=${HIVE_HOME:-/opt/apache-hive-3.1.2-bin}
export FLINK_HOME=${FLINK_HOME:-/opt/soft/flink}
export DATAX_LAUNCHER=${DATAX_LAUNCHER:-/opt/soft/datax/bin/python3}
export PATH=$HADOOP_HOME/bin:$SPARK_HOME/bin:$PYTHON_LAUNCHER:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_LAUNCHER:$PATH

2. 修改 install_env.sh

vim install_env.sh
ips=${ips:-"10.15.250.196,10.15.250.202"}# Port of SSH protocol, default value is 22. For now we only support same port in all `ips` machine
# modify it if you use different ssh port
sshPort=${sshPort:-"22"}# A comma separated list of machine hostname or IP would be installed Master server, it
# must be a subset of configuration `ips`.
# Example for hostnames: masters="ds1,ds2", Example for IPs: masters="192.168.8.1,192.168.8.2"
masters=${masters:-"10.15.250.196,10.15.250.202"}# A comma separated list of machine <hostname>:<workerGroup> or <IP>:<workerGroup>.All hostname or IP must be a
# subset of configuration `ips`, And workerGroup have default value as `default`, but we recommend you declare behind the hosts
# Example for hostnames: workers="ds1:default,ds2:default,ds3:default", Example for IPs: workers="192.168.8.1:default,192.168.8.2:default,192.168.8.3:default"
workers=${workers:-"10.15.250.196:default,10.15.250.202:default"}
# A comma separated list of machine hostname or IP would be installed Alert server, it
# must be a subset of configuration `ips`.
# Example for hostname: alertServer="ds3", Example for IP: alertServer="192.168.8.3"
alertServer=${alertServer:-"10.15.250.196"}# A comma separated list of machine hostname or IP would be installed API server, it
# must be a subset of configuration `ips`.
# Example for hostname: apiServers="ds1", Example for IP: apiServers="192.168.8.1"
apiServers=${apiServers:-"10.15.250.202"}# The directory to install DolphinScheduler for all machine we config above. It will automatically be created by `install.sh` script if not exists.
# Do not set this configuration same as the current path (pwd). Do not add quotes to it if you using related path.
installPath=${installPath:-"/opt/dolphinscheduler"}# The user to deploy DolphinScheduler for all machine we config above. For now user must create by yourself before running `install.sh`
# script. The user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled than the root directory needs
# to be created by this user
deployUser=${deployUser:-"root"}# The root of zookeeper, for now DolphinScheduler default registry server is zookeeper.
zkRoot=${zkRoot:-"/dolphinscheduler"}

3. 修改zk阻塞直到连接成功的等待时间。

如果不修改会报错。

  1. zk链接超时 zookeeper connect timeout: 10.15.250.196:2181 10.15.250.202:2181 10.15.250.220:2181
  2. worker worker in work group default have not received the heartbeat

如何做?

  1. 在work,master,api-server。修改配置
root@hadoop100:/opt/apache-dolphinscheduler-3.1.9-bin# vim api-server/conf/application.yaml
root@hadoop100:/opt/apache-dolphinscheduler-3.1.9-bin# vim work-server/conf/application.yaml
root@hadoop100:/opt/apache-dolphinscheduler-3.1.9-bin# vim master-server/conf/application.yaml

在这里插入图片描述

五:初始化mysql。

在mysql中执行以下命令。
CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'dolphinscheduler'@'%' IDENTIFIED BY 'dolphinscheduler';
GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'dolphinscheduler'@'10.15.250.196' IDENTIFIED BY 'dolphinscheduler';
flush privileges
cd /opt/apache-dolphinscheduler-3.1.9-bin
bash tools/bin/upgrade-schema.sh

六:进行初始化。

cd /opt/apache-dolphinscheduler-3.1.9-bin
bash ./bin/install.sh

七:查看页面。

因为所高可用,所以在俩个节点上进行查看。

地址
http://192.168.50.61:12345/dolphinscheduler/ui/login
http://192.168.50.60:12345/dolphinscheduler/ui/login
账号密码:admin/dolphinscheduler123

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.ryyt.cn/news/13.html

如若内容造成侵权/违法违规/事实不符,请联系我们进行投诉反馈,一经查实,立即删除!

相关文章

《TCP/IP详解 卷一》第7章 防火墙和NAT

7.1 引言 NAT通常改变源IP和源端口&#xff0c;不改变目的IP和目的端口。 7.2 防火墙 常用防火墙&#xff1a; 包过滤防火墙&#xff08;packet-filter firewall&#xff09; 代理防火墙&#xff08;proxy firewall&#xff09; 代理防火墙作用&#xff1a; 1. 通过代理服务…

2024图像处理分析与信息工程国际学术会议(IACIPIE2024)

2024图像处理分析与信息工程国际学术会议(IACIPIE2024) 会议简介 2024图像处理分析与信息工程国际学术会议&#xff08;IACIPIE2024&#xff09;将在中国长沙举行。 IACIPIE2024是一个年度会议&#xff0c;探讨图像处理分析和信息工程相关领域的发展和影响&#xff0c;旨在介…