分享

HiveUDF异常FAILED: Execution Error, return code 1 from org.apache.hadoop.hive...

nextuser 发表于 2016-11-14 14:34:00 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 1 11248



[mw_shl_code=bash,true]
Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"sourceid":"4649475029B8","sourcelabelw":"[{\"Name\":\"XX\",\"Weight\":6},{\"Name\":\"XX\",\"Weight\":6}]"}
        at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"sourceid":"4649475029B8","sourcelw":"[{\"Name\":\"XX\",\"Weight\":6},{\"Name\":\"XXX\",\"Weight\":6}]"}
        at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:550)
        at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
        ... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public java.lang.String cn.siti.main.MergeUDF.evaluate(java.lang.String,java.lang.String)  on object cn.siti.main.MergeUDF@9fd062 of class cn.siti.main.MergeUDF with arguments {[{"Name":"XXX","Weight":6},{"Name":"XXX","Weight":6}]:java.lang.String, XXXX,7:java.lang.String} of size 2
        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:1243)
        at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(GenericUDFBridge.java:182)
        at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
        at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
        at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
        at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79)
        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
        at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.internalForward(CommonJoinOperator.java:638)
        at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:670)
        at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:754)
        at org.apache.hadoop.hive.ql.exec.MapJoinOperator.processOp(MapJoinOperator.java:242)
        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
        at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
        at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:793)
        at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:540)
        ... 9 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:1219)
        ... 23 more
Caused by: com.alibaba.fastjson.JSONException: default constructor not found. class cn.siti.domain.TagInfo
        at com.alibaba.fastjson.util.JavaBeanInfo.build(JavaBeanInfo.java:212)
        at com.alibaba.fastjson.parser.ParserConfig.createJavaBeanDeserializer(ParserConfig.java:486)
        at com.alibaba.fastjson.parser.ParserConfig.getDeserializer(ParserConfig.java:433)
        at com.alibaba.fastjson.parser.ParserConfig.getDeserializer(ParserConfig.java:288)
        at com.alibaba.fastjson.parser.DefaultJSONParser.parseArray(DefaultJSONParser.java:651)
        at com.alibaba.fastjson.parser.DefaultJSONParser.parseArray(DefaultJSONParser.java:630)
        at com.alibaba.fastjson.parser.DefaultJSONParser.parseArray(DefaultJSONParser.java:625)
        at com.alibaba.fastjson.JSON.parseArray(JSON.java:491)
        at cn.siti.main.MergeUDF.evaluate(MergeUDF.java:21)
        ... 28 more[/mw_shl_code]
在使用的过程中请注意这一点,Hive中报错的时候,先看日志,从上往下找最终导致异常的地方,然后进行解决。

此外有时候会遇到一个很无奈的问题

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask   

There are 3 datanode(s) running and no node(s) are excluded in this operation  


本次运行中出现了这样的问题,我这边造成问题的主要原因是:nodemanager停止了,然后任务分发的时候出现问题。
重新启动yarn即可


出处:Hive数据仓库--HiveUDF异常解析与其中的坑
作者:低调的洋仔

已有(1)人评论

跳转到指定楼层
zhuqitian 发表于 2016-11-15 17:01:52
这个是自己虚拟机玩才会爆出的错吧
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条