Win主机+Linux虚拟机搭建Kettle集群

环境说明

程序 版本

Win:OS win 7

Linux:OS Redhat 6.2

JDK 1.8.0_45

Kettle 5.1

本机:Vmnet1 192.168.52.1

集群Master： 192.168.52.1

集群Slave1： 192.168.52.128

集群Slave2： 192.168.52.1

需要在Redhat 6.2上安装好jdk，配置好环境变量，并且把kettle安装好。（kettle不需要启动spoon来启动图形界面，利用kitchen调试好即可，linux上部署kettle，解压即可，因为kettle是跨平台运行的）。

注意：在集群运行环境中，需要端口打开或者直接关闭防火墙，供主服务器及子服务器互相连通通讯，在Redhat 6.2中，防火墙采用iptables的方式开启和关闭，检查端口是否开通，可以通过本机进行telnet测试。

首先，配置kettle主目录下的文件夹pwd中的carte-config-master-8080.xml、carte-config-8081.xml和carte-config-8082.xml文件，配置文件见下图：

注意：配置文件中的username和password并不是指主机的登陆账号和密码，是集群的账号密码，该账号密码是集群连接的依据，账号密码是通过混淆的方式保存在pwd文件，kettle默认的账号密码是cluster/cluster，修改该账号密码需要修改pwd文件，该文件也在pwd目录中，密码做了混淆。

Tips：

Kettle的集群搭建还是比较简单的，但是简单也有坑。

坑1：基于主机+虚拟机搭建集群的时候，一定要注意虚拟机与主机的网络通信方式，同时要考虑win主机的hosts文件的内容修改；

坑2：官网下载的kettle5或者6的主目录下是没有安装集群所必须的必要文件，可以去该地址下载相应文件：

Maven Central Repository Search

http://repo1.maven.org/maven2/org/owasp/esapi/esapi/2.1.0.1/

集群必要文件列表：

ESAPI.properties

validation.properties

antisamy-esapi.xml

集群必要jar文件：

esapi-2.1.0.1.jar

这个jar包记得放在lib文件夹里。

下面把配置步骤做如下描述：

一. MASTER 配置

进到pwd目录，打开文件carte-config-master-8080.xml，内容如下

<slave_config>

<!--

Document description...

 

- masters: You can list the slave servers to which this slave has to report back to.

If this is a master, we will contact the other masters to get a list of all the slaves in the cluster.

- report_to_masters : send a message to the defined masters to let them know we exist (Y/N)

- slaveserver : specify the slave server details of this carte instance.

IMPORTANT : the username and password specified here are used by the master instances to connect to this slave.

-->

<slaveserver>

<name>master1</name>

<hostname>192.168.52.1</hostname>

<port>8080</port>

<master>Y</master>

</slaveserver>

</slave_config>

Tips：这里注意<hostname>的value值选择本机的Vmnet1地址，因为我的主机与虚拟机是通过host-only方式通信的，如果你的虚拟机是通过NAT方式通信，那么此时的value要填入Vmnet8地址，原因自己想，理由很简单。

此时MASTER配置完毕。

二. SLAVE2 配置

进到pwd目录，打开文件carte-config-8082.xml，内容如下

<slave_config>

<!--

Document description...

 

- masters: You can list the slave servers to which this slave has to report back to.

If this is a master, we will contact the other masters to get a list of all the slaves in the cluster.

- report_to_masters : send a message to the defined masters to let them know we exist (Y/N)

- slaveserver : specify the slave server details of this carte instance.

IMPORTANT : the username and password specified here are used by the master instances to connect to this slave.

-->

<masters>

<slaveserver>

<name>master1</name>

<hostname>192.168.52.1</hostname>

<port>8080</port>

<username>cluster</username>

<password>cluster</password>

<master>Y</master>

</slaveserver>

</masters>

<report_to_masters>Y</report_to_masters>

<slaveserver>

<name>slave2-8082</name>

<hostname>192.168.52.1</hostname>

<port>8082</port>

<username>cluster</username>

<password>cluster</password>

<master>N</master>

</slaveserver>

</slave_config>

Tips：

子服务器比主服务器的配置内容多一点，该子服务器其实是与MASTER配置在一台主机上，除了自身的配置信息以外，要把MASTER的配置信息也配置好，因为启动子服务器的时候，子服务器需要根据配置文件中的MASTER地址和端口去MASTER主机注册服务，这里要注意的是配置子服务器的地方有个K-V配置，即<master>N</master>，如果是配置MASTER，这个位置的值是Y，如果配置的是子服务器，那么这个位置的值是N。

三. SLAVE3 配置

进到pwd目录，打开文件carte-config-8081.xml，内容如下

<slave_config>

<!--

Document description...

- masters: You can list the slave servers to which this slave has to report back to.

If this is a master, we will contact the other masters to get a list of all the slaves in the cluster.

- report_to_masters : send a message to the defined masters to let them know we exist (Y/N)

- slaveserver : specify the slave server details of this carte instance.

IMPORTANT : the username and password specified here are used by the master instances to connect to this slave.

-->

<masters>

<slaveserver>

<name>master1</name>

<hostname>192.168.52.1</hostname>

<port>8080</port>

<username>cluster</username>

<password>cluster</password>

<master>Y</master>

</slaveserver>

</masters>

<report_to_masters>Y</report_to_masters>

<slaveserver>

<name>slave1-8081</name>

<hostname>192.168.52.128</hostname>

<port>8081</port>

<username>cluster</username>

<password>cluster</password>

<master>N</master>

Tips：slave3是一台linux虚拟机，特别要注意在配置MASTER时的hostname的值，同时还要切记将前文提到的三个必要文件放在kettle的主目录中，否则会报错，报错内容如下：

Attempting to load validation.properties via file I/O.

Attempting to load validation.properties as resource file via file I/O.

Not found in "org.owasp.esapi.resources" directory or file not readable: /opt/data-integration-5.1/validation.properties

Not found in SystemResource Directory/resourceDirectory: .esapi/validation.properties

Not found in "user.home" (/root) directory: /root/esapi/validation.properties

Loading validation.properties via file I/O failed.

Attempting to load validation.properties via the classpath.

validation.properties could not be loaded by any means. fail. Exception was: java.lang.IllegalArgumentException: Failed to load ESAPI.properties as a classloader resource.

启动MASTER主机的carte服务：

D:softwaredata-integration-5.1>Carte.bat pwdcarte-config-master-8080.xml

DEBUG: Using JAVA_HOME

DEBUG: _PENTAHO_JAVA_HOME=D:jdk1.8.0_74

DEBUG: _PENTAHO_JAVA=D:jdk1.8.0_74injava.exe

2017/12/27 16:31:33 - Carte - Installing timer to purge stale objects after 1440

minutes.

2017-12-27 16:31:33.331::INFO: Logging to STDERR via org.mortbay.log.StdErrLog

2017/12/27 16:31:33 - Carte - 创建 web 服务监听器 @ 地址: 192.168.52.1:8080

2017-12-27 16:31:33.433::INFO: jetty-6.1.21

2017-12-27 16:31:33.458::INFO: Started SocketConnector@192.168.52.1:8080

Attempting to load ESAPI.properties via file I/O.

Attempting to load ESAPI.properties as resource file via file I/O.

Not found in "org.owasp.esapi.resources" directory or file not readable: D:soft

waredata-integration-5.1ESAPI.properties

Not found in SystemResource Directory/resourceDirectory: .esapiESAPI.properties

Not found in "user.home" (C:UsersAdministrator) directory: C:UsersAdministra

toresapiESAPI.properties

Loading ESAPI.properties via file I/O failed. Exception was: java.io.FileNotFoun

dException

Attempting to load ESAPI.properties via the classpath.

SUCCESSFULLY LOADED ESAPI.properties via the CLASSPATH from "/ (root)" using cur

rent thread context class loader!

Attempting to load validation.properties via file I/O.

Attempting to load validation.properties as resource file via file I/O.

Not found in "org.owasp.esapi.resources" directory or file not readable: D:soft

waredata-integration-5.1validation.properties

Not found in SystemResource Directory/resourceDirectory: .esapivalidation.prope

rties

Not found in "user.home" (C:UsersAdministrator) directory: C:UsersAdministra

toresapivalidation.properties

Loading validation.properties via file I/O failed.

Attempting to load validation.properties via the classpath.

SUCCESSFULLY LOADED validation.properties via the CLASSPATH from "/ (root)" usin

g current thread context class loader!

打开浏览器，输入地址：http://192.168.52.1:8080

输入用户名和密码(就是在配置文件中配置的用户名和密码)

Tips:注意右下角，有提示这是MASTER.

启动slave1

[root@racnode2 data-integration-5.1]# carte.sh pwd/carte-config-8081.xml

/opt/data-integration-5.1

2017/12/27 00:34:21 - Carte - Installing timer to purge stale objects after 1440 minutes.

System property [org.owasp.esapi.opsteam] is not set

System property [org.owasp.esapi.devteam] is not set

Attempting to load ESAPI.properties via file I/O.

Attempting to load ESAPI.properties as resource file via file I/O.

Not found in "org.owasp.esapi.resources" directory or file not readable: /opt/data-integration-5.1/ESAPI.properties

Not found in SystemResource Directory/resourceDirectory: .esapi/ESAPI.properties

Not found in "user.home" (/root) directory: /root/esapi/ESAPI.properties

Loading ESAPI.properties via file I/O failed. Exception was: java.io.FileNotFoundException

Attempting to load ESAPI.properties via the classpath.

SUCCESSFULLY LOADED ESAPI.properties via the CLASSPATH from "/ (root)" using current thread context class loader!

SecurityConfiguration for Validator.ConfigurationFile.MultiValued not found in ESAPI.properties. Using default: false

Attempting to load validation.properties via file I/O.

Attempting to load validation.properties as resource file via file I/O.

Not found in "org.owasp.esapi.resources" directory or file not readable: /opt/data-integration-5.1/validation.properties

Not found in SystemResource Directory/resourceDirectory: .esapi/validation.properties

Not found in "user.home" (/root) directory: /root/esapi/validation.properties

Loading validation.properties via file I/O failed.

Attempting to load validation.properties via the classpath.

SUCCESSFULLY LOADED validation.properties via the CLASSPATH from "/ (root)" using current thread context class loader!

2017/12/27 00:34:21 - Carte - Registered this slave server to master slave server [master1] on address [192.168.52.1:8080]

2017/12/27 00:34:21 - Carte - Registered this slave server to master slave server [master1] on address [192.168.52.1:8080]

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

SLF4J: Defaulting to no-operation (NOP) logger implementation

SLF4J: See SLF4J Error Codes for further details.

2017/12/27 00:34:22 - Carte - Created listener for webserver @ address : 192.168.52.128:8081

打开浏览器，输入地址：http://192.168.52.128:8081

启动slave2

D:softwaredata-integration-5.1>Carte.bat pwdcarte-config-8082.xml

DEBUG: Using JAVA_HOME

DEBUG: _PENTAHO_JAVA_HOME=D:jdk1.8.0_74

DEBUG: _PENTAHO_JAVA=D:jdk1.8.0_74injava.exe

2017/12/27 16:38:09 - Carte - Installing timer to purge stale objects after 1440

minutes.

Attempting to load ESAPI.properties via file I/O.

Attempting to load ESAPI.properties as resource file via file I/O.

Not found in "org.owasp.esapi.resources" directory or file not readable: D:soft

waredata-integration-5.1ESAPI.properties

Not found in SystemResource Directory/resourceDirectory: .esapiESAPI.properties

Not found in "user.home" (C:UsersAdministrator) directory: C:UsersAdministra

toresapiESAPI.properties

Loading ESAPI.properties via file I/O failed. Exception was: java.io.FileNotFoun

dException

Attempting to load ESAPI.properties via the classpath.

SUCCESSFULLY LOADED ESAPI.properties via the CLASSPATH from "/ (root)" using cur

rent thread context class loader!

Attempting to load validation.properties via file I/O.

Attempting to load validation.properties as resource file via file I/O.

Not found in "org.owasp.esapi.resources" directory or file not readable: D:soft

waredata-integration-5.1validation.properties

Not found in SystemResource Directory/resourceDirectory: .esapivalidation.prope

rties

Not found in "user.home" (C:UsersAdministrator) directory: C:UsersAdministra

toresapivalidation.properties

Loading validation.properties via file I/O failed.

Attempting to load validation.properties via the classpath.

SUCCESSFULLY LOADED validation.properties via the CLASSPATH from "/ (root)" usin

g current thread context class loader!

2017/12/27 16:38:09 - Carte - Registered this slave server to master slave serve

r [master1] on address [192.168.52.1:8080]

2017/12/27 16:38:09 - Carte - Registered this slave server to master slave serve

r [master1] on address [192.168.52.1:8080]

2017-12-27 16:38:09.398::INFO: Logging to STDERR via org.mortbay.log.StdErrLog

2017/12/27 16:38:09 - Carte - 创建 web 服务监听器 @ 地址: 192.168.52.1:8082

2017-12-27 16:38:09.456::INFO: jetty-6.1.21

2017-12-27 16:38:09.475::INFO: Started SocketConnector@192.168.52.1:8082

打开浏览器，输入地址：http://192.168.52.1:8082

关于在kettle开发过程中转换和工作流里的集群配置，以后再写个文章说明。————————————————

End.

爱数据网作者：Joe

作者介绍：通信和金融行业高级数据建模工程师 

本文为挖数网作者原创文章，未经允许禁止转载，需要转载请微信联系授权（微信号：lovedata0520）

更多文章前往爱数据社区网站首页浏览http://www.itongji.cn/