azkaban集群多节点模式配置

配置多节点执行服务器的时候,需要在AzkabanWebServer的配置文件azkaban.properties里添加

azkaban.use.multiple.executors=true azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1 azkaban.executorselector.comparator.Memory=1 azkaban.executorselector.comparator.LastDispatched=1 azkaban.executorselector.comparator.CpuUsage=1 

以确认使用的是分布式方式,随后提交的job会根据情况自行选择执行服务器,否则默认只使用本地执行服务器,切记!

之后需要在mysql的azkaban库中添加各个执行服务器的ip/域名和端口:

insert into executors(host,port) values("EXECUTOR_PORT",EXECUTOR_PORT);

至此,azkaban的分布式配置才算成功。

配置多节点执行服务器的时候,需要在AzkabanWebServer的配置文件azkaban.properties里添加

azkaban.use.multiple.executors=true azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1 azkaban.executorselector.comparator.Memory=1 azkaban.executorselector.comparator.LastDispatched=1 azkaban.executorselector.comparator.CpuUsage=1 

以确认使用的是分布式方式,随后提交的job会根据情况自行选择执行服务器,否则默认只使用本地执行服务器,切记!

之后需要在mysql的azkaban库中添加各个执行服务器的ip/域名和端口:

insert into executors(host,port) values("EXECUTOR_PORT",EXECUTOR_PORT);

至此,azkaban的分布式配置才算成功。


以下来源官网文档: http://azkaban.github.io/azkaban/docs/latest/

The Executor server needs to be setup with a port, and the AzabanWebServer will need to know what this port is.

The following properties need to be set on AzkabanExecutorServer's azkaban.properties.

# Azkaban Executor settings executor.maxThreads=50 executor.port=12321 executor.flow.threads=30 

By default the executor.port is set to 12321. The AzkabanWebServer will have to point to this port as well.

This is done by setting the following property in AzkabanWebServer's azkaban.properties.

executor.port=12321 

We need to enable multiple executor mode in webserver configuration if we want to run in multiple executor mode. Please verify that you have following properties in azkaban.properties. azkaban.use.multiple.executors and azkaban.executorselector.comparator.* are required properties. Please note that azkaban.use.multiple.executors is not honoured in multiple exeutor mode.

azkaban.use.multiple.executors=true azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1 azkaban.executorselector.comparator.Memory=1 azkaban.executorselector.comparator.LastDispatched=1 azkaban.executorselector.comparator.CpuUsage=1 

These changes are only picked up after restarting the servers.

We don't have a executor management UI at this point of Time. Executors need to be configured in database. For examples:-

Insert all executors into the mysql DB for executor setups. Verify that correct executors are active in executors table.insert into executors(host,port) values("EXECUTOR_PORT",EXECUTOR_PORT);

原文链接:https://blog.csdn.net/jacklin929/article/details/53742638

原创文章,作者:优速盾-小U,如若转载,请注明出处:https://www.cdnb.net/bbs/archives/6924

(0)
上一篇 2022年10月24日 02:37
下一篇 2022年10月24日

相关推荐

发表回复

您的电子邮箱地址不会被公开。 必填项已用*标注

优速盾注册领取大礼包www.cdnb.net
/sitemap.xml