15/03/09 20:13:38 WARN scheduler.TaskSetManager: Lost task 202.0 in stage 1.0 (TID 217, slave19.cluster03): java.io.IOException: com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 13994
Serialization trace:
ord (org.apache.spark.util.BoundedPriorityQueue)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1003)
at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:164)
at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:87)
at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:61)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 13994
Serialization trace:
ord (org.apache.spark.util.BoundedPriorityQueue)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)
at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610)
at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:599)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732)
at org.apache.spark.serializer.KryoDeserializationStream.readObject(KryoSerializer.scala:142)
at org.apache.spark.broadcast.TorrentBroadcast$.unBlockifyObject(TorrentBroadcast.scala:216)
at org.apache.spark.broadcast.TorrentBroadcast$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:177)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1000)
... 12 more
在spark-default.conf中我将spark.serializer配置成org.apache.spark.serializer.KryoSerializer,但并没有正确的配置,改回org.apache.spark.serializer.JavaSerializer即可,关于如何正确配置成org.apache.spark.serializer.KryoSerializer日后会介绍~