分享

hive可用,但是执行数据量大时就报错

yangyixin 发表于 2017-8-9 18:31:36 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 2 18144
用hive查询1g数据量时就报错,这个错误代码都不知道去哪里找:Stack trace: ExitCodeException exitCode=255:
但是此时的hive能使用,select count(*) from pagecount;都可以查询。
然后减小数据量到400m试了一下,发现能用,请问这是怎么回事,要怎么处理啊?
谢谢啦

hive报错如下:
hive> SELECT percentile(countall,array(0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9)) FROM pagecount;
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
Query ID = spark2_20170809190519_17a24e4c-a303-4d87-ae3a-879f3cfcf8b4
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Job = job_1501586770689_0094, Tracking URL = http://hop51:8088/proxy/application_1501586770689_0094/
Kill Command = /home/user/spark2/apps/hadoop-2.7.3/bin/hadoop job  -kill job_1501586770689_0094
Hadoop job information for Stage-1: number of mappers: 5; number of reducers: 1
2017-08-09 19:05:29,517 Stage-1 map = 0%,  reduce = 0%
2017-08-09 19:05:37,832 Stage-1 map = 40%,  reduce = 0%, Cumulative CPU 21.58 sec
2017-08-09 19:05:38,874 Stage-1 map = 60%,  reduce = 0%, Cumulative CPU 34.32 sec
2017-08-09 19:05:39,907 Stage-1 map = 93%,  reduce = 0%, Cumulative CPU 71.9 sec
2017-08-09 19:05:40,939 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 72.77 sec
2017-08-09 19:06:41,952 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 72.77 sec
2017-08-09 19:07:42,807 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 72.77 sec
2017-08-09 19:08:05,522 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 72.77 sec
MapReduce Total cumulative CPU time: 1 minutes 12 seconds 770 msec
Ended Job = job_1501586770689_0094 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1501586770689_0094_m_000003 (and more) from job job_1501586770689_0094

Task with the most failures(4):
-----
Task ID:
  task_1501586770689_0094_r_000000

URL:
  http://hop51:8088/taskdetails.jsp?jobid=job_1501586770689_0094&tipid=task_1501586770689_0094_r_000000
-----
Diagnostic Messages for this Task:
Exception from container-launch.
Container id: container_1501586770689_0094_01_000010
Exit code: 255
Stack trace: ExitCodeException exitCode=255:
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:582)
    at org.apache.hadoop.util.Shell.run(Shell.java:479)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:773)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 255


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 5  Reduce: 1   Cumulative CPU: 72.77 sec   HDFS Read: 875790735 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 1 minutes 12 seconds 770 msec

已有(2)人评论

跳转到指定楼层
langke93 发表于 2017-8-9 19:13:28
增大内存试试例如[mw_shl_code=bash,true]<property>
   <name>mapred.child.java.opts</name>
   <value>-Xmx1024m</value>
</property> [/mw_shl_code]
hive中修改如下
set io.sort.mb =400;

来自:跟楼主遇到的问题类似,数据量一大就有问题
hive sql 执行错误
http://www.aboutyun.com/forum.php?mod=viewthread&tid=22303




回复

使用道具 举报

yangyixin 发表于 2017-8-15 16:40:13
我没有修改这条,因为不知道在哪里修改,
hive中修改如下
set io.sort.mb =400;
按照大神的说法,修改了
[mw_shl_code=applescript,true]<property>
   <name>mapred.child.java.opts</name>
   <value>-Xmx1024m</value>
</property>[/mw_shl_code]
这条是要配合yarn-site.xml修改的。
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条