分享

map慢,有啥优化空间,可以调哪些参数

ananan36 发表于 2015-5-29 15:16:45 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 6 16970
hive> select 1 from filelog;      
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1432881085436_0004, Tracking URL = http://0.0.0.0:23355/proxy/application_1432881085436_0004/
Kill Command = /hdfs/data01/hadoop-2.5.0-cdh5.2.0/bin/hadoop job  -kill job_1432881085436_0004
Hadoop job information for Stage-1: number of mappers: 229; number of reducers: 0
2015-05-29 15:04:20,837 Stage-1 map = 0%,  reduce = 0%
2015-05-29 15:04:39,402 Stage-1 map = 1%,  reduce = 0%, Cumulative CPU 9.51 sec
2015-05-29 15:04:44,587 Stage-1 map = 2%,  reduce = 0%, Cumulative CPU 19.36 sec
2015-05-29 15:04:50,776 Stage-1 map = 3%,  reduce = 0%, Cumulative CPU 30.27 sec
2015-05-29 15:04:56,959 Stage-1 map = 4%,  reduce = 0%, Cumulative CPU 45.32 sec
2015-05-29 15:04:59,030 Stage-1 map = 5%,  reduce = 0%, Cumulative CPU 55.28 sec
2015-05-29 15:05:04,183 Stage-1 map = 6%,  reduce = 0%, Cumulative CPU 64.4 sec
2015-05-29 15:05:09,349 Stage-1 map = 7%,  reduce = 0%, Cumulative CPU 74.68 sec
2015-05-29 15:05:15,519 Stage-1 map = 8%,  reduce = 0%, Cumulative CPU 93.05 sec
2015-05-29 15:05:19,651 Stage-1 map = 9%,  reduce = 0%, Cumulative CPU 98.09 sec
2015-05-29 15:05:22,735 Stage-1 map = 10%,  reduce = 0%, Cumulative CPU 107.34 sec
2015-05-29 15:05:27,872 Stage-1 map = 11%,  reduce = 0%, Cumulative CPU 120.83 sec
2015-05-29 15:05:33,011 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 129.47 sec
2015-05-29 15:05:38,149 Stage-1 map = 13%,  reduce = 0%, Cumulative CPU 138.84 sec
2015-05-29 15:05:40,201 Stage-1 map = 14%,  reduce = 0%, Cumulative CPU 147.36 sec
2015-05-29 15:05:46,368 Stage-1 map = 15%,  reduce = 0%, Cumulative CPU 160.56 sec
2015-05-29 15:05:50,473 Stage-1 map = 16%,  reduce = 0%, Cumulative CPU 169.67 sec
2015-05-29 15:05:56,629 Stage-1 map = 17%,  reduce = 0%, Cumulative CPU 182.85 sec
2015-05-29 15:06:01,785 Stage-1 map = 18%,  reduce = 0%, Cumulative CPU 191.67 sec
2015-05-29 15:06:03,835 Stage-1 map = 19%,  reduce = 0%, Cumulative CPU 200.02 sec
2015-05-29 15:06:08,964 Stage-1 map = 20%,  reduce = 0%, Cumulative CPU 220.57 sec
2015-05-29 15:06:15,120 Stage-1 map = 21%,  reduce = 0%, Cumulative CPU 233.74 sec
2015-05-29 15:06:21,275 Stage-1 map = 22%,  reduce = 0%, Cumulative CPU 246.78 sec
2015-05-29 15:06:22,309 Stage-1 map = 23%,  reduce = 0%, Cumulative CPU 251.09 sec
2015-05-29 15:06:27,431 Stage-1 map = 24%,  reduce = 0%, Cumulative CPU 264.02 sec
2015-05-29 15:06:33,580 Stage-1 map = 25%,  reduce = 0%, Cumulative CPU 276.97 sec
2015-05-29 15:06:38,718 Stage-1 map = 26%,  reduce = 0%, Cumulative CPU 281.05 sec
2015-05-29 15:06:40,765 Stage-1 map = 27%,  reduce = 0%, Cumulative CPU 289.63 sec
2015-05-29 15:06:45,896 Stage-1 map = 28%,  reduce = 0%, Cumulative CPU 298.12 sec
2015-05-29 15:06:52,045 Stage-1 map = 29%,  reduce = 0%, Cumulative CPU 311.12 sec
2015-05-29 15:06:57,172 Stage-1 map = 30%,  reduce = 0%, Cumulative CPU 319.89 sec
2015-05-29 15:06:59,222 Stage-1 map = 31%,  reduce = 0%, Cumulative CPU 329.38 sec
2015-05-29 15:07:04,347 Stage-1 map = 32%,  reduce = 0%, Cumulative CPU 342.47 sec
2015-05-29 15:07:10,493 Stage-1 map = 33%,  reduce = 0%, Cumulative CPU 355.5 sec
2015-05-29 15:07:14,598 Stage-1 map = 34%,  reduce = 0%, Cumulative CPU 359.65 sec
2015-05-29 15:07:21,767 Stage-1 map = 35%,  reduce = 0%, Cumulative CPU 383.61 sec
2015-05-29 15:07:23,813 Stage-1 map = 36%,  reduce = 0%, Cumulative CPU 388.1 sec

已有(6)人评论

跳转到指定楼层
NEOGX 发表于 2015-5-29 22:44:56
hive优化方法很多
楼主这个可以对查询的字段创建索引,这个可以变快

具体操作,参考:
Hive如何创建索引

#############################

更多优化:
hive优化以及执行原理

hive 全排序优化

Hive 查询优化总结

深入浅出数据仓库中SQL性能优化之Hive篇
回复

使用道具 举报

ananan36 发表于 2015-6-1 08:49:34
如何提升map性能
回复

使用道具 举报

tntzbzc 发表于 2015-6-3 19:40:45
ananan36 发表于 2015-6-1 08:49
如何提升map性能

没有单纯绝对的优化,都是根据自己集群的情况,
参考这个:
控制hive任务中的map数
回复

使用道具 举报

ananan36 发表于 2015-6-4 08:55:56
tntzbzc 发表于 2015-6-3 19:40
没有单纯绝对的优化,都是根据自己集群的情况,
参考这个:
控制hive任务中的map数

这个是map慢,能从mapreduce去优化么?比如调整copy fetch,shuffle参数吗
回复

使用道具 举报

bob007 发表于 2015-6-5 15:56:13
ananan36 发表于 2015-6-4 08:55
这个是map慢,能从mapreduce去优化么?比如调整copy fetch,shuffle参数吗

优化的方式有多种,楼主可以多尝试,资料也很多。

回复

使用道具 举报

yepiaochen 发表于 2015-6-8 13:45:22
可以优化Map,优化Hql,最好先找下资料吧...
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条