Win主机+Linux虚拟机搭建Kettle集群

爱数据精选
爱数据精选
爱数据精选
609
文章
0
评论
2021-12-1814:52:55 评论 412 10346字
摘要

Kettle集群的搭建

环境说明

程序 版本

Win:OS win 7

Linux:OS Redhat 6.2

JDK 1.8.0_45

Kettle 5.1

本机:Vmnet1 192.168.52.1

集群Master: 192.168.52.1

集群Slave1: 192.168.52.128

集群Slave2: 192.168.52.1

需要在Redhat 6.2上安装好jdk,配置好环境变量,并且把kettle安装好。(kettle不需要启动spoon来启动图形界面,利用kitchen调试好即可,linux上部署kettle,解压即可,因为kettle是跨平台运行的)。

注意:在集群运行环境中,需要端口打开或者直接关闭防火墙,供主服务器及子服务器互相连通通讯,在Redhat 6.2中,防火墙采用iptables的方式开启和关闭,检查端口是否开通,可以通过本机进行telnet测试。

首先,配置kettle主目录下的文件夹pwd中的carte-config-master-8080.xml、carte-config-8081.xml和carte-config-8082.xml文件,配置文件见下图:

注意:配置文件中的username和password并不是指主机的登陆账号和密码,是集群的账号密码,该账号密码是集群连接的依据,账号密码是通过混淆的方式保存在pwd文件,kettle默认的账号密码是cluster/cluster,修改该账号密码需要修改pwd文件,该文件也在pwd目录中,密码做了混淆。

Tips:

Kettle的集群搭建还是比较简单的,但是简单也有坑。

坑1:基于主机+虚拟机搭建集群的时候,一定要注意虚拟机与主机的网络通信方式,同时要考虑win主机的hosts文件的内容修改;

坑2:官网下载的kettle5或者6的主目录下是没有安装集群所必须的必要文件,可以去该地址下载相应文件:

Maven Central Repository Search

http://repo1.maven.org/maven2/org/owasp/esapi/esapi/2.1.0.1/

集群必要文件列表:

ESAPI.properties

validation.properties

antisamy-esapi.xml

集群必要jar文件:

esapi-2.1.0.1.jar

这个jar包记得放在lib文件夹里。

下面把配置步骤做如下描述:

一. MASTER 配置

进到pwd目录,打开文件carte-config-master-8080.xml,内容如下

<slave_config>

<!--

Document description...

 

- masters: You can list the slave servers to which this slave has to report back to.

If this is a master, we will contact the other masters to get a list of all the slaves in the cluster.

- report_to_masters : send a message to the defined masters to let them know we exist (Y/N)

- slaveserver : specify the slave server details of this carte instance.

IMPORTANT : the username and password specified here are used by the master instances to connect to this slave.

-->

<slaveserver>

<name>master1</name>

<hostname>192.168.52.1</hostname>

<port>8080</port>

<master>Y</master>

</slaveserver>

</slave_config>

Tips:这里注意<hostname>的value值选择本机的Vmnet1地址,因为我的主机与虚拟机是通过host-only方式通信的,如果你的虚拟机是通过NAT方式通信,那么此时的value要填入Vmnet8地址,原因自己想,理由很简单。

此时MASTER配置完毕。

二. SLAVE2 配置

进到pwd目录,打开文件carte-config-8082.xml,内容如下

<slave_config>

<!--

Document description...

 

- masters: You can list the slave servers to which this slave has to report back to.

If this is a master, we will contact the other masters to get a list of all the slaves in the cluster.

- report_to_masters : send a message to the defined masters to let them know we exist (Y/N)

- slaveserver : specify the slave server details of this carte instance.

IMPORTANT : the username and password specified here are used by the master instances to connect to this slave.

-->

<masters>

<slaveserver>

<name>master1</name>

<hostname>192.168.52.1</hostname>

<port>8080</port>

<username>cluster</username>

<password>cluster</password>

<master>Y</master>

</slaveserver>

</masters>

<report_to_masters>Y</report_to_masters>

<slaveserver>

<name>slave2-8082</name>

<hostname>192.168.52.1</hostname>

<port>8082</port>

<username>cluster</username>

<password>cluster</password>

<master>N</master>

</slaveserver>

</slave_config>

Tips:

子服务器比主服务器的配置内容多一点,该子服务器其实是与MASTER配置在一台主机上,除了自身的配置信息以外,要把MASTER的配置信息也配置好,因为启动子服务器的时候,子服务器需要根据配置文件中的MASTER地址和端口去MASTER主机注册服务,这里要注意的是配置子服务器的地方有个K-V配置,即<master>N</master>,如果是配置MASTER,这个位置的值是Y,如果配置的是子服务器,那么这个位置的值是N。

三. SLAVE3 配置

进到pwd目录,打开文件carte-config-8081.xml,内容如下

<slave_config>

<!--

Document description...

- masters: You can list the slave servers to which this slave has to report back to.

If this is a master, we will contact the other masters to get a list of all the slaves in the cluster.

- report_to_masters : send a message to the defined masters to let them know we exist (Y/N)

- slaveserver : specify the slave server details of this carte instance.

IMPORTANT : the username and password specified here are used by the master instances to connect to this slave.

-->

<masters>

<slaveserver>

<name>master1</name>

<hostname>192.168.52.1</hostname>

<port>8080</port>

<username>cluster</username>

<password>cluster</password>

<master>Y</master>

</slaveserver>

</masters>

<report_to_masters>Y</report_to_masters>

<slaveserver>

<name>slave1-8081</name>

<hostname>192.168.52.128</hostname>

<port>8081</port>

<username>cluster</username>

<password>cluster</password>

<master>N</master>

Tips:slave3是一台linux虚拟机,特别要注意在配置MASTER时的hostname的值,同时还要切记将前文提到的三个必要文件放在kettle的主目录中,否则会报错,报错内容如下:

Attempting to load validation.properties via file I/O.

Attempting to load validation.properties as resource file via file I/O.

Not found in "org.owasp.esapi.resources" directory or file not readable: /opt/data-integration-5.1/validation.properties

Not found in SystemResource Directory/resourceDirectory: .esapi/validation.properties

Not found in "user.home" (/root) directory: /root/esapi/validation.properties

Loading validation.properties via file I/O failed.

Attempting to load validation.properties via the classpath.

validation.properties could not be loaded by any means. fail. Exception was: java.lang.IllegalArgumentException: Failed to load ESAPI.properties as a classloader resource.

启动MASTER主机的carte服务:

D:softwaredata-integration-5.1>Carte.bat pwdcarte-config-master-8080.xml

DEBUG: Using JAVA_HOME

DEBUG: _PENTAHO_JAVA_HOME=D:jdk1.8.0_74

DEBUG: _PENTAHO_JAVA=D:jdk1.8.0_74injava.exe

2017/12/27 16:31:33 - Carte - Installing timer to purge stale objects after 1440

minutes.

2017-12-27 16:31:33.331::INFO: Logging to STDERR via org.mortbay.log.StdErrLog

2017/12/27 16:31:33 - Carte - 创建 web 服务监听器 @ 地址: 192.168.52.1:8080

2017-12-27 16:31:33.433::INFO: jetty-6.1.21

2017-12-27 16:31:33.458::INFO: Started SocketConnector@192.168.52.1:8080

Attempting to load ESAPI.properties via file I/O.

Attempting to load ESAPI.properties as resource file via file I/O.

Not found in "org.owasp.esapi.resources" directory or file not readable: D:soft

waredata-integration-5.1ESAPI.properties

Not found in SystemResource Directory/resourceDirectory: .esapiESAPI.properties

Not found in "user.home" (C:UsersAdministrator) directory: C:UsersAdministra

toresapiESAPI.properties

Loading ESAPI.properties via file I/O failed. Exception was: java.io.FileNotFoun

dException

Attempting to load ESAPI.properties via the classpath.

SUCCESSFULLY LOADED ESAPI.properties via the CLASSPATH from "/ (root)" using cur

rent thread context class loader!

Attempting to load validation.properties via file I/O.

Attempting to load validation.properties as resource file via file I/O.

Not found in "org.owasp.esapi.resources" directory or file not readable: D:soft

waredata-integration-5.1validation.properties

Not found in SystemResource Directory/resourceDirectory: .esapivalidation.prope

rties

Not found in "user.home" (C:UsersAdministrator) directory: C:UsersAdministra

toresapivalidation.properties

Loading validation.properties via file I/O failed.

Attempting to load validation.properties via the classpath.

SUCCESSFULLY LOADED validation.properties via the CLASSPATH from "/ (root)" usin

g current thread context class loader!

打开浏览器,输入地址:http://192.168.52.1:8080

输入用户名和密码(就是在配置文件中配置的用户名和密码)

Tips:注意右下角,有提示这是MASTER.

启动slave1

[root@racnode2 data-integration-5.1]# carte.sh pwd/carte-config-8081.xml

/opt/data-integration-5.1

2017/12/27 00:34:21 - Carte - Installing timer to purge stale objects after 1440 minutes.

System property [org.owasp.esapi.opsteam] is not set

System property [org.owasp.esapi.devteam] is not set

Attempting to load ESAPI.properties via file I/O.

Attempting to load ESAPI.properties as resource file via file I/O.

Not found in "org.owasp.esapi.resources" directory or file not readable: /opt/data-integration-5.1/ESAPI.properties

Not found in SystemResource Directory/resourceDirectory: .esapi/ESAPI.properties

Not found in "user.home" (/root) directory: /root/esapi/ESAPI.properties

Loading ESAPI.properties via file I/O failed. Exception was: java.io.FileNotFoundException

Attempting to load ESAPI.properties via the classpath.

SUCCESSFULLY LOADED ESAPI.properties via the CLASSPATH from "/ (root)" using current thread context class loader!

SecurityConfiguration for Validator.ConfigurationFile.MultiValued not found in ESAPI.properties. Using default: false

Attempting to load validation.properties via file I/O.

Attempting to load validation.properties as resource file via file I/O.

Not found in "org.owasp.esapi.resources" directory or file not readable: /opt/data-integration-5.1/validation.properties

Not found in SystemResource Directory/resourceDirectory: .esapi/validation.properties

Not found in "user.home" (/root) directory: /root/esapi/validation.properties

Loading validation.properties via file I/O failed.

Attempting to load validation.properties via the classpath.

SUCCESSFULLY LOADED validation.properties via the CLASSPATH from "/ (root)" using current thread context class loader!

2017/12/27 00:34:21 - Carte - Registered this slave server to master slave server [master1] on address [192.168.52.1:8080]

2017/12/27 00:34:21 - Carte - Registered this slave server to master slave server [master1] on address [192.168.52.1:8080]

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

SLF4J: Defaulting to no-operation (NOP) logger implementation

SLF4J: See SLF4J Error Codes for further details.

2017/12/27 00:34:22 - Carte - Created listener for webserver @ address : 192.168.52.128:8081

打开浏览器,输入地址:http://192.168.52.128:8081

启动slave2

D:softwaredata-integration-5.1>Carte.bat pwdcarte-config-8082.xml

DEBUG: Using JAVA_HOME

DEBUG: _PENTAHO_JAVA_HOME=D:jdk1.8.0_74

DEBUG: _PENTAHO_JAVA=D:jdk1.8.0_74injava.exe

2017/12/27 16:38:09 - Carte - Installing timer to purge stale objects after 1440

minutes.

Attempting to load ESAPI.properties via file I/O.

Attempting to load ESAPI.properties as resource file via file I/O.

Not found in "org.owasp.esapi.resources" directory or file not readable: D:soft

waredata-integration-5.1ESAPI.properties

Not found in SystemResource Directory/resourceDirectory: .esapiESAPI.properties

Not found in "user.home" (C:UsersAdministrator) directory: C:UsersAdministra

toresapiESAPI.properties

Loading ESAPI.properties via file I/O failed. Exception was: java.io.FileNotFoun

dException

Attempting to load ESAPI.properties via the classpath.

SUCCESSFULLY LOADED ESAPI.properties via the CLASSPATH from "/ (root)" using cur

rent thread context class loader!

Attempting to load validation.properties via file I/O.

Attempting to load validation.properties as resource file via file I/O.

Not found in "org.owasp.esapi.resources" directory or file not readable: D:soft

waredata-integration-5.1validation.properties

Not found in SystemResource Directory/resourceDirectory: .esapivalidation.prope

rties

Not found in "user.home" (C:UsersAdministrator) directory: C:UsersAdministra

toresapivalidation.properties

Loading validation.properties via file I/O failed.

Attempting to load validation.properties via the classpath.

SUCCESSFULLY LOADED validation.properties via the CLASSPATH from "/ (root)" usin

g current thread context class loader!

2017/12/27 16:38:09 - Carte - Registered this slave server to master slave serve

r [master1] on address [192.168.52.1:8080]

2017/12/27 16:38:09 - Carte - Registered this slave server to master slave serve

r [master1] on address [192.168.52.1:8080]

2017-12-27 16:38:09.398::INFO: Logging to STDERR via org.mortbay.log.StdErrLog

2017/12/27 16:38:09 - Carte - 创建 web 服务监听器 @ 地址: 192.168.52.1:8082

2017-12-27 16:38:09.456::INFO: jetty-6.1.21

2017-12-27 16:38:09.475::INFO: Started SocketConnector@192.168.52.1:8082

打开浏览器,输入地址:http://192.168.52.1:8082

关于在kettle开发过程中转换和工作流里的集群配置,以后再写个文章说明。————————————————

End.

爱数据网作者:Joe

作者介绍:通信和金融行业高级数据建模工程师 

本文为挖数网作者原创文章,未经允许禁止转载,需要转载请微信联系授权(微信号:lovedata0520)

更多文章前往爱数据社区网站首页浏览http://www.itongji.cn/

  • 我的微信公众号
  • 微信扫一扫
  • weinxin
  • 我的微信公众号
  • 微信扫一扫
  • weinxin
匿名

发表评论

匿名网友 填写信息

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: