DolphinScheduler配置Seatunnel同步任务出错(从MySQL同步数据到Doris)
DolphinSScheduler版本3.2.0, Seatunnel版本2.3.4
错误日志
定义好工作流后, 上线运行报错, 关键日志入下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 [INFO] 2025-02-16 13:02:00.427 -0800 - tenantCode :root, task dir:/tmp/dolphinscheduler/exec/process/root/16670700117984/16677825236448_2/2/3 [INFO] 2025-02-16 13:02:00.429 -0800 - generate script file:/tmp/dolphinscheduler/exec/process/root/16670700117984/16677825236448_2/2/3/seatunnel_2_3.conf [INFO] 2025-02-16 13:02:00.433 -0800 - SeaTunnel task command: ${SEATUNNEL_HOME}/bin/seatunnel.sh --config /tmp/dolphinscheduler/exec/process/root/16670700117984/16677825236448_2/2/3/seatunnel_2_3.conf --deploy-mode local [INFO] 2025-02-16 13:02:00.436 -0800 - Final Shell file is : #!/bin/bash BASEDIR=$(cd `dirname $0`; pwd) cd $BASEDIR ${SEATUNNEL_HOME}/bin/seatunnel.sh --config /tmp/dolphinscheduler/exec/process/root/16670700117984/16677825236448_2/2/3/seatunnel_2_3.conf --deploy-mode local [INFO] 2025-02-16 13:02:00.439 -0800 - Executing shell command : sudo -u root -i /tmp/dolphinscheduler/exec/process/root/16670700117984/16677825236448_2/2/3/2_3.sh [INFO] 2025-02-16 13:02:00.483 -0800 - process start, process id is: 37662 [INFO] 2025-02-16 13:02:02.484 -0800 - -> /tmp/dolphinscheduler/exec/process/root/16670700117984/16677825236448_2/2/3/2_3.sh: line 4: /bin/seatunnel.sh: No such file or directory [INFO] 2025-02-16 13:02:02.491 -0800 - process has exited. execute path:/tmp/dolphinscheduler/exec/process/root/16670700117984/16677825236448_2/2/3, processId:37662 ,exitStatusCode:127 ,processWaitForStatus:true ,processExitValue:127 [INFO] 2025-02-16 13:02:02.493 -0800 - *********************************************************************************************** [INFO] 2025-02-16 13:02:02.505 -0800 - ********************************* Finalize task instance ************************************ [INFO] 2025-02-16 13:02:02.506 -0800 - *********************************************************************************************** [INFO] 2025-02-16 13:02:02.509 -0800 - Upload output files: [] successfully [INFO] 2025-02-16 13:02:02.512 -0800 - Send task execute status: FAILURE to master : 192.168.10.132:1234 [INFO] 2025-02-16 13:02:02.514 -0800 - Remove the current task execute context from worker cache [INFO] 2025-02-16 13:02:02.515 -0800 - The current execute mode isn't develop mode, will clear the task execute file: /tmp/dolphinscheduler/exec/process/root/16670700117984/16677825236448_2/2/3 [INFO] 2025-02-16 13:02:02.519 -0800 - Success clear the task execute file: /tmp/dolphinscheduler/exec/process/root/16670700117984/16677825236448_2/2/3 [INFO] 2025-02-16 13:02:02.520 -0800 - FINALIZE_SESSION
解决尝试
先检查DolphinScheduler的Seatunnel环境变量是否配置正确
发现输出为空, 但我已经在dolphinscheduler_env.sh配置了export SEATUNNEL_HOME=${SEATUNNEL_HOME:-/opt/software/seatunnel-2.3.4}, 于是决定重启DolphinScheduler, 重启后再次尝试, 环境依旧未生效, 进入dolphinscheduler_env.sh所在目录, 手动加载文件
1 source dolphinscheduler_env.sh
手动加载后运行echo $SEATUNNEL_HOME
, 成功显示/opt/software/seatunnel-2.3.4, 再次重启DolphinScheduler, 运行依旧报错
意识到应该是我修改过的dolphinscheduler_env.sh文件没有生效, 应该重新执行install.sh脚本, 重新安装DolphinScheduler, 由于3.2.0是新的第一个大版本, 于是决定回退到3.1.9
重装Dolphinscheduler
停止ds服务
删除dolphinscheduler目录
1 rm -rf /opt/software/dolphinscheduler-3.2.0
删除ds元数据库
1 DROP DATABASE dolphinscheduler;
配置dolphinscheduler_env.sh, 根据自己的安装目录和服务器ip和账户密码配置
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 cd /opt/software/dolphinscheduler-3.1.9/bin/env vim dolphinscheduler_env.sh export JAVA_HOME=${JAVA_HOME:-/opt/software/openjdk8} # Database related configuration, set database type , username and password export DATABASE=${DATABASE:-mysql} export SPRING_PROFILES_ACTIVE=${DATABASE} export SPRING_DATASOURCE_URL=jdbc:mysql://ip:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&useSSL=false export SPRING_DATASOURCE_USERNAME=root export SPRING_DATASOURCE_PASSWORD= # DolphinScheduler server related configuration export SPRING_CACHE_TYPE=${SPRING_CACHE_TYPE:-none} export SPRING_JACKSON_TIME_ZONE=${SPRING_JACKSON_TIME_ZONE:-UTC} export MASTER_FETCH_COMMAND_NUM=${MASTER_FETCH_COMMAND_NUM:-10} # Registry center configuration, determines the type and link of the registry center export REGISTRY_TYPE=${REGISTRY_TYPE:-zookeeper} export REGISTRY_ZOOKEEPER_CONNECT_STRING=${REGISTRY_ZOOKEEPER_CONNECT_STRING:-ip:2181} # Tasks related configurations, need to change the configuration if you use the related tasks. export SEATUNNEL_HOME=${SEATUNNEL_HOME:-/opt/software/seatunnel-2.3.4} export PATH=$JAVA_HOME/bin:$SEATUNNEL_HOME/bin:$PATH
配置install_env.sh, 根据自己集群资源配置
1 2 3 4 5 6 7 8 9 10 11 12 cd /opt/software/dolphinscheduler-3.1.9/bin/env vim install_env.sh ips=${ips:-""} sshPort=${sshPort:-"22"} masters=${masters:-""} workers=${workers:-""} alertServer=${alertServer:-""} apiServers=${apiServers:-""} installPath=${installPath:-"/opt/software/dolphinscheduler-3.1.9/ds"} deployUser=${deployUser:-"root"} zkRoot=${zkRoot:-"/dolphinscheduler"}
连接mysql, 创建元数据库
1 2 3 4 5 6 CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL PRIVILEGES ON dolphinscheduler.* TO 'root' @'%' ; FLUSH PRIVILEGES;
执行 upgrade-schema.sh
1 2 3 4 5 cd /opt/software/dolphinscheduler-3.1.9 chmod 777 tools/bin/upgrade-schema.sh bash tools/bin/upgrade-schema.sh
创建dolphinscheduler 安装目录, 这是由于我在install_env.sh中的配置, 要根据自己的情况来
1 2 cd /opt/software/dolphinscheduler-3.1.9/ mkdir ds
执行install.sh
1 2 cd /opt/software/dolphinscheduler-3.1.9/bin/ ./install.sh
使用默认账密admin/dolphinscheduler123登陆
检查配置是否生效
重新进行同步任务, 成功.