困惑了一周的hive 查询 ，mapreduce 报错问题，续

hive 版本 1.2.1 hadoop 版本 2.6.3

主机名称	JOURNALNODE	NAMENODE	DATANODE	YARN	SPARK	HIVE	MYSQL
Server1-1	JOURNALNODE				MASTER	CLIENT
Server1-3			DATANODE	NodeManager	WORKER
Server2-1	JOURNALNODE	NAMENODE		ResourceManager
Server2-3			DATANODE	NodeManager	WORKER
Server3-1	JOURNALNODE	NAMENODE				SERVER	MYSQL
Server3-3			DATANODE	NodeManager	WORKER

所有nodemanage 复制了 hive(不确定是否 必要)。
经过一天的系统重新设置，日志查看，困惑的问题一直存在。但偶尔能够执行成功一次。HIVE 执行 select count(*) from table 提示错误
hive> select count(*) from logstable limit 10;
Query ID = netuser1_20160129093334_f73a817f-b6f5-43e9-8a68-fa42d99f6a90
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Job = job_1454031176192_0001, Tracking URL = http://server2-
1:8088/proxy/application_1454031176192_0001/
Kill Command = /dbc/hadoop-2.6.3/bin/hadoop job  -kill job_1454031176192_0001
Hadoop job information for Stage-1: number of mappers: 0; number of reducers: 0
2016-01-29 09:33:44,118 Stage-1 map = 0%,  reduce = 0%
Ended Job = job_1454031176192_0001 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1:  HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
yarn只提示了一个jar包错误，其他未提示。nodemanage上有大量jar包本地化失败的日志

再次查看nodemanage日志记录。发现每次执行hive 查询的同时，nodemanage都要复制hive ,lib 下所有的jar包，进行本地化。

2016-01-31 01:36:08,899 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource file:/dbc/hive-1.2.1/lib/commons-io-2.4.jar transitioned from INIT to DOWNLOADING
2016-01-31 01:36:09,165 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Downloading public rsrc:{ file:/dbc/hive-1.2.1/lib/commons-io-2.4.jar, 1453963763000, FILE, null }
本地化所有的包，正常就是这种日志。
2016-01-31 01:36:12,639 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource file:/dbc/hive-1.2.1/lib/commons-io-2.4.jar(->/dbc/hadoop-2.6.3/yarn/local/filecache/104/commons-io-2.4.jar) transitioned from DOWNLOADING to LOCALIZED


执行错误就是这种日志
2016-01-30 21:50:20,410 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource file:/dbc/hive-1.2.1/lib/commons-io-2.4.jar transitioned from INIT to DOWNLOADING
2016-01-30 21:50:20,498 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Downloading public rsrc:{ file:/dbc/hive-1.2.1/lib/commons-io-2.4.jar, 1453953223000, FILE, null }
2016-01-30 21:50:20,584 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Failed to download resource { { file:/dbc/hive-1.2.1/lib/commons-io-2.4.jar, 1453953223000, FILE, null },pending,[],1081500033828265,DOWNLOADING}java.io.IOException: Resource file:/dbc/hive-1.2.1/lib/commons-io-2.4.jar changed on src filesystem (expected 1453953223000, was 14539637630002016-01-30 21:50:20,598 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalizedResource: Resource file:/dbc/hive-1.2.1/lib/commons-io-2.4.jar(->/dbc/hadoop-2.6.3/yarn/local/filecache/214/commons-io-2.4.jar) transitioned from DOWNLOADING to FAILED2016-01-30 21:50:20,598 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Failed to download resource { { file:/dbc/hive-1.2.1/lib/json-20090211.jar, 1453953223000, FILE, null },pending,[],1081500033832761,DOWNLOADING}java.io.IOException: Resource file:/dbc/hive-1.2.1/lib/json-20090211.jar changed on src filesystem (expected 1453953223000, was 1453963763000


查看mapreduce filecache目录如下：
每台nodemanage filecache 目录中存在本地化的hive jar 包 ，目录序号 至 118 ，但提示错误的本地化过程提示的目录号并不存在。

[netuser1@server1-3 filecache]$ pwd
/dbc/hadoop-2.6.3/yarn/local/filecache
[netuser1@server1-3 filecache]$ dir
10   102  105  108  110  113  116  119  14  17  20  23  26  29  32  35  38  41  44  47  50  53  56  59  62  65  68  71  74  77  80  83  86  89  92  95  98
100  103  106  109  111  114  117  12   15  18  21  24  27  30  33  36  39  42  45  48  51  54  57  60  63  66  69  72  75  78  81  84  87  90  93  96  99
101  104  107  11   112  115  118  13   16  19  22  25  28  31  34  37  40  43  46  49  52  55  58  61  64  67  70  73  76  79  82  85  88  91  94  97

分析，每次nodemanage 本地化加载hive jar 包，并本地创建filecache ，提示错误的本地化过程，本地并未创建filecache目录 （见日志中粗体字编号）。

问题：
1、为什么每次都要复制这些文件至本地？
2、hive 安装与nodemanage 的关系是什么？（yarn 是否需要每台nodemanage都复制hive）

这是楼主配置不正确造成的，下面配置了吗？
HADOOP_CONF_DIR or YARN_CONF_DIR
配置正确了，应该就不会发生这种情况了

楼主hadoop是否有旧的包，我觉得吧，这肯定是个死胡同了。楼主重新搭建，会解决这个问题，甚至可能不会遇到这个问题。
个人认为还是版本混乱或则其它jar包造成的

hadoop hive整合还是比较简单的，楼主可以尝试搭建一个稳定版本的整合，最新版本初次学习，很可能搞不定


补充内容 (2016-2-2 17:17):
网上很多的，先找足资料，在安装

按照这个试试
hadoop2.2+hbase0.96+hive0.12安装整合详细高可靠文档及经验总结
http://www.aboutyun.com/thread-7890-1-1.html

这个问题最后找到原因了没？？？我也碰到了？？？！！！！

ysyzww 发表于 2016-12-7 10:15
这个问题最后找到原因了没？？？我也碰到了？？？！！！！

两个问题：
1.可能是内存的问题：包括jvm
2.版本兼容的问题