分享

hive sql异常:Encountered unregistered class ID,求解决???

bbhhhh 发表于 2015-4-27 11:21:29 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 3 51465
本帖最后由 bbhhhh 于 2015-4-27 11:31 编辑

环境是hadoop2.5-cdh5.2.1, hive0.13.1-cdh5.2.
有20个hive sql需要执行,不是每次都报错,即使报错,也不是发生在固定的sql上,论坛里看到有同学也碰到,但是没有看到解决办法,求大神指点。
2015-04-27 10:41:33,449 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2015-04-27 10:41:33,756 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: Paths:/tmp/hive-hadoop/hive_2015-04-27_10-39-59_371_8227897410315895740-1/-mr-10004/000000_0:0+189InputFormatClass: org.apache.hadoop.mapred.SequenceFileInputFormat

2015-04-27 10:41:33,829 INFO [main] org.apache.hadoop.hive.ql.log.PerfLogger: <PERFLOG method=deserializePlan from=org.apache.hadoop.hive.ql.exec.Utilities>
2015-04-27 10:41:33,829 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: Deserializing MapWork via kryo
2015-04-27 10:41:34,053 ERROR [main] org.apache.hadoop.hive.ql.exec.Utilities: Failed to load plan: hdfs://c7000-master:8020/tmp/hive-hadoop/hive_2015-04-27_10-39-59_371_8227897410315895740-5/-mr-10020/9e94cf5d-1841-49fc-ab4f-45e11ade550c/map.xml
org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: -243410070

Serialization trace:
startTimes (org.apache.hadoop.hive.ql.log.PerfLogger)
perfLogger (org.apache.hadoop.hive.ql.exec.MapJoinOperator)
childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
        at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)
        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)
        at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:99)
        at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
        at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
        at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
        at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
        at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
        at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
        at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
        at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
        at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
        at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
        at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
        at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
        at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:918)
        at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:826)
        at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:840)
        at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:333)
        at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:275)
        at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:254)
        at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:440)
        at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:433)
        at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
        at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:169)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)



已有(3)人评论

跳转到指定楼层
qz2003 发表于 2015-4-27 12:30:51
hive 在执行前,都有sql执行计划,

加载失败,是下面配置可能出了问题
hive.exec.scratchdir  
HDFS路径,用于存储不同 map/reduce 阶段的执行计划和这些阶段的中间输出结果。  
默认值:/tmp/<user.name>/hive



所以楼主到这个路径下看看

另外查看hive执行计划
explain+sql

比如:

explain SELECT a.bar, count(*) FROM invites a WHERE a.foo > 0 GROUP BY a.bar;


更多内容:








回复

使用道具 举报

s060403072 发表于 2015-4-27 15:39:12
是否测试过5条或则10条
可能因为执行过多造成不稳定
回复

使用道具 举报

leletuo2012 发表于 2016-2-22 10:38:49
你找到这个问题的解决办法了么,我现在也遇到了 。据说Kyro对map支持的不是很好
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条