环境:hadoop3.1.1 +hive3.1.1 + zookeeper3.4.13
问题:
我给Yarn配置了HA,连续执行两条Hive会产生报错,然而不适用Yarn的HA就没有这个问题,求高手解答。。。
在一台机器上启动了metastore,然后启动hive客户端
出现如下问题:
第一条正常
hive> select count(*) from normal_table;
再次执行报错
hive> select count(*) from normal_table;Query ID = ctu_20190106000653_451f5d82-301b-4030-80b6-ce30e8a0cd0c
Total jobs = 1
Launching Job 1 out of 1
FAILED: Hive Internal Error: java.lang.IllegalArgumentException(The value of property yarn.resourcemanager.zk-address must not be null)
java.lang.IllegalArgumentException: The value of property yarn.resourcemanager.zk-address must not be null
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:146)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.initialize(ExecDriver.java:181)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2649)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2335)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:218)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
我的YarnHA配置经过检查并未发现错误,相关配置如下:
<!-- HA -->
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>ctu-yarn</value>
</property>
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>ctu-hdp-01</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>ctu-hdp-02</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm1</name>
<value>ctu-hdp-01:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address.rm2</name>
<value>ctu-hdp-02:8088</value>
</property>
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>ctu-hdp-03:2181,ctu-hdp-04:2181,ctu-hdp-05:2181</value>
</property>
补充内容 (2019-1-6 14:57):
刚才重新编译代码加日志输出看了一下,在执行完第一次select之后yarn.resourcemanager.zk-address就已经被清空了。。。 |