分享

hive运行一段时间后异常,变成map = 0%, reduce = 0%

aurae 发表于 2015-10-13 09:33:23 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 5 61665
有个hive任务,执行这个sql
insert overwrite table all select * from tmp union all select a.* from all a left outer join tmp b on a.key_=b.key_ where b.key_ is null;
all表的字段很多,大概有100+。而且记录行数大概有15000万条记录,tmp表是大概1000万的记录。
这个sql执行了大概8个小时后,日志信息变成了这个样子,查看userlogs,没有发现错误日志。

Query ID = hadoop_20151012180720_63799a5c-bb6b-4540-9456-99e9cf4b3055
Total jobs = 4
Stage-9 is selected by condition resolver.
Launching Job 1 out of 4
Number of reduce tasks not specified. Estimated from input data size: 1009
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Job = job_1443504148509_1391, Tracking URL = http://zhebuduan-bd-3:8088/proxy/application_1443504148509_1391/
Kill Command = /data/hadoop/bin/hadoop job  -kill job_1443504148509_1391
Hadoop job information for Stage-9: number of mappers: 635; number of reducers: 1009
2015-10-12 18:07:37,247 Stage-9 map = 0%,  reduce = 0%
2015-10-12 18:08:38,147 Stage-9 map = 0%,  reduce = 0%, Cumulative CPU 203.49 sec
2015-10-12 18:09:03,796 Stage-9 map = 1%,  reduce = 0%, Cumulative CPU 528.45 sec
2015-10-12 18:09:52,068 Stage-9 map = 2%,  reduce = 0%, Cumulative CPU 1019.9 sec
2015-10-12 18:10:25,827 Stage-9 map = 3%,  reduce = 0%, Cumulative CPU 1385.3 sec
2015-10-12 18:10:56,393 Stage-9 map = 4%,  reduce = 0%, Cumulative CPU 1703.76 sec
2015-10-12 18:11:34,268 Stage-9 map = 5%,  reduce = 0%, Cumulative CPU 2004.47 sec
2015-10-12 18:12:26,060 Stage-9 map = 6%,  reduce = 0%, Cumulative CPU 2403.02 sec
2015-10-12 18:13:04,377 Stage-9 map = 7%,  reduce = 0%, Cumulative CPU 2702.93 sec
2015-10-12 18:13:43,170 Stage-9 map = 8%,  reduce = 0%, Cumulative CPU 2911.85 sec
2015-10-12 18:14:43,375 Stage-9 map = 8%,  reduce = 0%, Cumulative CPU 3154.9 sec
2015-10-12 18:14:50,515 Stage-9 map = 9%,  reduce = 0%, Cumulative CPU 3184.58 sec
2015-10-12 18:15:51,401 Stage-9 map = 9%,  reduce = 0%, Cumulative CPU 3498.33 sec
2015-10-12 18:16:14,622 Stage-9 map = 10%,  reduce = 0%, Cumulative CPU 3601.82 sec
2015-10-12 18:17:13,091 Stage-9 map = 11%,  reduce = 0%, Cumulative CPU 3877.44 sec
2015-10-12 18:18:13,291 Stage-9 map = 11%,  reduce = 0%, Cumulative CPU 4305.78 sec
2015-10-12 18:18:14,674 Stage-9 map = 12%,  reduce = 0%, Cumulative CPU 4321.6 sec
2015-10-12 18:18:53,776 Stage-9 map = 13%,  reduce = 0%, Cumulative CPU 4627.11 sec
2015-10-12 18:19:49,960 Stage-9 map = 14%,  reduce = 0%, Cumulative CPU 4891.36 sec
2015-10-12 18:20:41,203 Stage-9 map = 15%,  reduce = 0%, Cumulative CPU 5227.76 sec
2015-10-12 18:21:30,959 Stage-9 map = 16%,  reduce = 0%, Cumulative CPU 5454.73 sec
2015-10-12 18:22:31,348 Stage-9 map = 16%,  reduce = 0%, Cumulative CPU 5743.29 sec
2015-10-12 18:22:49,822 Stage-9 map = 17%,  reduce = 0%, Cumulative CPU 5873.21 sec
2015-10-12 18:23:31,626 Stage-9 map = 18%,  reduce = 0%, Cumulative CPU 6209.85 sec
2015-10-12 18:24:29,372 Stage-9 map = 19%,  reduce = 0%, Cumulative CPU 6590.27 sec
2015-10-12 18:25:06,385 Stage-9 map = 20%,  reduce = 0%, Cumulative CPU 6874.18 sec
2015-10-12 18:26:07,484 Stage-9 map = 21%,  reduce = 0%, Cumulative CPU 7317.21 sec
。。。
2015-10-12 19:37:34,843 Stage-9 map = 99%,  reduce = 1%, Cumulative CPU 29639.04 sec
2015-10-12 19:38:35,279 Stage-9 map = 99%,  reduce = 1%, Cumulative CPU 29852.16 sec
2015-10-12 19:39:35,613 Stage-9 map = 99%,  reduce = 1%, Cumulative CPU 29970.1 sec
2015-10-12 19:40:35,623 Stage-9 map = 99%,  reduce = 1%, Cumulative CPU 30073.87 sec
2015-10-12 19:41:36,712 Stage-9 map = 99%,  reduce = 1%, Cumulative CPU 30139.35 sec
2015-10-12 19:42:36,838 Stage-9 map = 99%,  reduce = 1%, Cumulative CPU 30184.23 sec
2015-10-12 19:42:40,271 Stage-9 map = 100%,  reduce = 2%, Cumulative CPU 30190.43 sec
2015-10-12 19:42:42,918 Stage-9 map = 100%,  reduce = 3%, Cumulative CPU 30204.1 sec
2015-10-12 19:43:43,579 Stage-9 map = 100%,  reduce = 3%, Cumulative CPU 30794.97 sec
2015-10-12 19:44:02,425 Stage-9 map = 100%,  reduce = 4%, Cumulative CPU 30988.25 sec
2015-10-12 19:45:03,904 Stage-9 map = 100%,  reduce = 4%, Cumulative CPU 31429.73 sec
2015-10-12 19:45:38,526 Stage-9 map = 100%,  reduce = 5%, Cumulative CPU 31592.5 sec
2015-10-12 19:46:39,369 Stage-9 map = 100%,  reduce = 5%, Cumulative CPU 31929.99 sec
2015-10-12 19:47:25,419 Stage-9 map = 100%,  reduce = 6%, Cumulative CPU 32143.89 sec
。。。
2015-10-12 22:15:17,288 Stage-9 map = 100%,  reduce = 92%, Cumulative CPU 73779.76 sec
2015-10-12 22:16:05,995 Stage-9 map = 100%,  reduce = 93%, Cumulative CPU 74042.81 sec
2015-10-12 22:17:06,205 Stage-9 map = 100%,  reduce = 93%, Cumulative CPU 74345.85 sec
2015-10-12 22:17:37,457 Stage-9 map = 100%,  reduce = 94%, Cumulative CPU 74486.9 sec
2015-10-12 22:18:37,747 Stage-9 map = 100%,  reduce = 94%, Cumulative CPU 74808.58 sec
2015-10-12 22:19:05,703 Stage-9 map = 100%,  reduce = 95%, Cumulative CPU 74979.29 sec
2015-10-12 22:20:06,006 Stage-9 map = 100%,  reduce = 95%, Cumulative CPU 75300.08 sec
2015-10-12 22:20:32,823 Stage-9 map = 86%,  reduce = 96%, Cumulative CPU 71438.63 sec
2015-10-12 22:20:35,958 Stage-9 map = 86%,  reduce = 95%, Cumulative CPU 71131.66 sec
2015-10-12 22:20:37,044 Stage-9 map = 86%,  reduce = 94%, Cumulative CPU 70932.32 sec
2015-10-12 22:21:38,217 Stage-9 map = 86%,  reduce = 94%, Cumulative CPU 71167.7 sec
2015-10-12 22:22:38,990 Stage-9 map = 86%,  reduce = 94%, Cumulative CPU 71513.1 sec
2015-10-12 22:23:03,081 Stage-9 map = 87%,  reduce = 94%, Cumulative CPU 71649.71 sec
2015-10-12 22:23:06,904 Stage-9 map = 87%,  reduce = 95%, Cumulative CPU 71665.3 sec
2015-10-12 22:24:03,535 Stage-9 map = 88%,  reduce = 95%, Cumulative CPU 71995.52 sec
2015-10-12 22:24:44,591 Stage-9 map = 89%,  reduce = 95%, Cumulative CPU 72212.29 sec
2015-10-12 22:25:45,107 Stage-9 map = 89%,  reduce = 95%, Cumulative CPU 72548.04 sec
2015-10-12 22:25:47,660 Stage-9 map = 90%,  reduce = 95%, Cumulative CPU 72566.11 sec
2015-10-12 22:26:32,838 Stage-9 map = 79%,  reduce = 95%, Cumulative CPU 69230.88 sec
2015-10-12 22:26:34,031 Stage-9 map = 72%,  reduce = 94%, Cumulative CPU 66730.61 sec
2015-10-12 22:26:38,429 Stage-9 map = 51%,  reduce = 94%, Cumulative CPU 62813.45 sec
2015-10-12 22:26:51,324 Stage-9 map = 43%,  reduce = 94%, Cumulative CPU 59955.35 sec
2015-10-12 22:27:51,701 Stage-9 map = 43%,  reduce = 94%, Cumulative CPU 60084.47 sec
2015-10-12 22:28:52,187 Stage-9 map = 43%,  reduce = 94%, Cumulative CPU 60271.56 sec
2015-10-12 22:28:56,515 Stage-9 map = 44%,  reduce = 94%, Cumulative CPU 60287.18 sec
。。。
2015-10-13 00:22:18,406 Stage-9 map = 96%,  reduce = 94%, Cumulative CPU 76767.26 sec
2015-10-13 00:22:44,277 Stage-9 map = 97%,  reduce = 94%, Cumulative CPU 76838.81 sec
2015-10-13 00:23:44,483 Stage-9 map = 97%,  reduce = 94%, Cumulative CPU 76975.32 sec
2015-10-13 00:24:44,828 Stage-9 map = 97%,  reduce = 94%, Cumulative CPU 77126.26 sec
2015-10-13 00:25:10,766 Stage-9 map = 98%,  reduce = 94%, Cumulative CPU 77193.25 sec
2015-10-13 00:26:11,175 Stage-9 map = 98%,  reduce = 94%, Cumulative CPU 77351.77 sec
2015-10-13 00:27:11,706 Stage-9 map = 98%,  reduce = 94%, Cumulative CPU 77495.48 sec
2015-10-13 00:27:33,215 Stage-9 map = 99%,  reduce = 94%, Cumulative CPU 77550.39 sec
2015-10-13 00:28:33,279 Stage-9 map = 99%,  reduce = 94%, Cumulative CPU 77663.3 sec
2015-10-13 00:29:33,549 Stage-9 map = 99%,  reduce = 94%, Cumulative CPU 77822.39 sec
2015-10-13 00:30:33,740 Stage-9 map = 99%,  reduce = 94%, Cumulative CPU 77931.9 sec
2015-10-13 00:31:32,820 Stage-9 map = 100%,  reduce = 94%, Cumulative CPU 78011.91 sec
2015-10-13 00:31:37,117 Stage-9 map = 100%,  reduce = 95%, Cumulative CPU 78017.0 sec
2015-10-13 00:32:40,952 Stage-9 map = 100%,  reduce = 95%, Cumulative CPU 78252.01 sec
2015-10-13 00:32:42,042 Stage-9 map = 54%,  reduce = 95%, Cumulative CPU 62335.43 sec
2015-10-13 00:32:44,180 Stage-9 map = 54%,  reduce = 94%, Cumulative CPU 61986.19 sec
2015-10-13 00:33:44,448 Stage-9 map = 54%,  reduce = 94%, Cumulative CPU 62001.12 sec
2015-10-13 00:34:35,394 Stage-9 map = 55%,  reduce = 94%, Cumulative CPU 62049.79 sec
。。。
2015-10-13 01:56:36,291 Stage-9 map = 92%,  reduce = 94%, Cumulative CPU 72734.36 sec
2015-10-13 01:57:37,050 Stage-9 map = 92%,  reduce = 94%, Cumulative CPU 72854.83 sec
2015-10-13 01:58:37,715 Stage-9 map = 92%,  reduce = 94%, Cumulative CPU 72975.54 sec
2015-10-13 01:58:38,795 Stage-9 map = 93%,  reduce = 94%, Cumulative CPU 72985.46 sec
2015-10-13 01:59:39,648 Stage-9 map = 93%,  reduce = 94%, Cumulative CPU 73117.88 sec
2015-10-13 02:00:36,277 Stage-9 map = 94%,  reduce = 94%, Cumulative CPU 73232.79 sec
2015-10-13 02:01:36,984 Stage-9 map = 94%,  reduce = 94%, Cumulative CPU 73372.9 sec
2015-10-13 02:02:35,425 Stage-9 map = 95%,  reduce = 94%, Cumulative CPU 73525.49 sec
2015-10-13 02:03:36,000 Stage-9 map = 95%,  reduce = 94%, Cumulative CPU 73644.4 sec
2015-10-13 02:04:31,343 Stage-9 map = 0%,  reduce = 0%
2015-10-13 02:05:31,797 Stage-9 map = 0%,  reduce = 0%
2015-10-13 02:06:32,344 Stage-9 map = 0%,  reduce = 0%
2015-10-13 02:07:32,610 Stage-9 map = 0%,  reduce = 0%
2015-10-13 02:08:32,938 Stage-9 map = 0%,  reduce = 0%
2015-10-13 02:09:33,339 Stage-9 map = 0%,  reduce = 0%
2015-10-13 02:10:33,750 Stage-9 map = 0%,  reduce = 0%
2015-10-13 02:11:34,621 Stage-9 map = 0%,  reduce = 0%
2015-10-13 02:12:35,505 Stage-9 map = 0%,  reduce = 0%
2015-10-13 02:13:36,460 Stage-9 map = 0%,  reduce = 0%
2015-10-13 02:14:36,479 Stage-9 map = 0%,  reduce = 0%, Cumulative CPU 97.59 sec
2015-10-13 02:15:36,892 Stage-9 map = 0%,  reduce = 0%, Cumulative CPU 187.14 sec
2015-10-13 02:16:31,215 Stage-9 map = 1%,  reduce = 0%, Cumulative CPU 244.9 sec
2015-10-13 02:16:32,387 Stage-9 map = 0%,  reduce = 0%
2015-10-13 02:17:32,912 Stage-9 map = 0%,  reduce = 0%
2015-10-13 02:18:33,562 Stage-9 map = 0%,  reduce = 0%
然后就一直这样打印map = 0%, reduce = 0%了,这是不是hadoop的一个bug呀。有明白的大神帮忙科普一下,谢谢拉

已有(5)人评论

跳转到指定楼层
Alkaloid0515 发表于 2015-10-13 10:36:23
初步判断hive数据倾斜

hive数据倾斜原因分析及解决方案
http://www.aboutyun.com/thread-8296-1-1.html



Hive 数据倾斜总结
http://www.aboutyun.com/thread-8241-1-1.html


hive卡住不动,并非数据倾斜,这个是什么原因
http://www.aboutyun.com/thread-10037-1-1.html



回复

使用道具 举报

aurae 发表于 2015-10-16 10:38:24
非常感谢,那我从数据倾斜的方面找找原因看看
回复

使用道具 举报

miedongdong 发表于 2017-3-24 14:28:40
你好,请问是什么原因,您找到了么,我是刚接触,得接手整个项目,暂时找不着原因。如能回复,万分感激
回复

使用道具 举报

starrycheng 发表于 2017-3-24 15:03:18
miedongdong 发表于 2017-3-24 14:28
你好,请问是什么原因,您找到了么,我是刚接触,得接手整个项目,暂时找不着原因。如能回复,万分感激

用这个命令看下,数据是否分布均匀
hadoop dfsadmin -report 命令详解
有其它问题可开贴问,看到人更多
回复

使用道具 举报

miedongdong 发表于 2017-3-25 17:09:52
starrycheng 发表于 2017-3-24 15:03
用这个命令看下,数据是否分布均匀
hadoop dfsadmin -report 命令详解
有其它问题可开贴问,看到人更多 ...

我们的好像是内存不够的问题,现在问题暂时没有重现。谢了
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条