Hadoop启动异常：UnregisteredDatanodeException

启动Hadoop集群测，发现三台DataNode只启动成功了两台，未启动成功的那一台日志中出现了下列异常：

hadoop-root-datanode-hadoop2.log  hadoop-root-datanode-hadoop2.out  hadoop-root-tasktracker-hadoop2.log  hadoop-root-tasktracker-hadoop2.out
You have new mail in /var/spool/mail/root
[root@hadoop2 logs]# more  hadoop-root-datanode-hadoop2.log
2014-12-10 17:41:13,323 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG:   host = hadoop2/192.168.56.102
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 1.1.2
STARTUP_MSG:   build = https://svn.apache.org/repos/asf ... branches/branch-1.1 -r 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
************************************************************/
2014-12-10 17:41:13,999 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2014-12-10 17:41:14,051 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2014-12-10 17:41:14,054 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2014-12-10 17:41:14,054 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started
2014-12-10 17:41:14,559 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2014-12-10 17:41:19,445 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Registered FSDatasetStatusMBean
2014-12-10 17:41:19,516 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened data transfer server at 50010
2014-12-10 17:41:19,534 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith is 1048576 bytes/s
2014-12-10 17:41:20,224 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2014-12-10 17:41:20,971 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2014-12-10 17:41:21,123 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dfs.webhdfs.enabled = false
2014-12-10 17:41:21,123 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075
2014-12-10 17:41:21,124 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075
2014-12-10 17:41:21,124 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075
2014-12-10 17:41:21,129 INFO org.mortbay.log: jetty-6.1.26
2014-12-10 17:41:23,820 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50075
2014-12-10 17:41:23,842 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
2014-12-10 17:41:23,849 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source DataNode registered.
2014-12-10 17:41:27,146 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
2014-12-10 17:41:27,170 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort50020 registered.
2014-12-10 17:41:27,175 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort50020 registered.
2014-12-10 17:41:27,211 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnRegistration = DatanodeRegistration(hadoop2:50010, storageID=DS-834248845-192.168.56.102-50010-1415256060466,
infoPort=50075, ipcPort=50020)
2014-12-10 17:41:27,235 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Finished generating blocks being written report for 1 volumes in 0 seconds
2014-12-10 17:41:27,254 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting asynchronous block report scan
2014-12-10 17:41:27,263 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Finished asynchronous block report scan in 9ms
2014-12-10 17:41:27,263 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(192.168.56.102:50010, storageID=DS-834248845-192.168.56.102-50010-1415256060466, infoPort=
50075, ipcPort=50020)In DataNode.run, data = FSDataset{dirpath='/usr/local/hadoop/datanode/current'}
2014-12-10 17:41:27,272 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2014-12-10 17:41:27,286 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 50020: starting
2014-12-10 17:41:27,289 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 50020: starting
2014-12-10 17:41:27,294 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: using BLOCKREPORT_INTERVAL of 3600000msec Initial delay: 0msec
2014-12-10 17:41:27,296 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
2014-12-10 17:41:27,300 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 50020: starting
2014-12-10 17:41:27,314 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Reconciled asynchronous block report against current state in 3 ms
2014-12-10 17:41:27,335 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 15 blocks took 6 msec to generate and 19 msecs for RPC and NN processing
2014-12-10 17:41:27,336 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting Periodic block scanner.
2014-12-10 17:41:27,343 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Generated rough (lockless) block report in 2 ms
2014-12-10 17:41:27,343 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Reconciled asynchronous block report against current state in 0 ms
2014-12-10 17:41:27,604 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2014-12-10 17:41:28,196 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification succeeded for blk_1604557343193261362_1014
2014-12-10 17:41:30,300 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action: DNA_REGISTER
2014-12-10 17:41:30,304 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Finished generating blocks being written report for 1 volumes in 0 seconds
2014-12-10 17:41:30,308 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting asynchronous block report scan
2014-12-10 17:41:30,309 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Finished asynchronous block report scan in 1ms
2014-12-10 17:41:33,296 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Reconciled asynchronous block report against current state in 0 ms
2014-12-10 17:41:33,314 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DataNode is shutting down: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.protocol.UnregisteredD
atanodeException: Data node 192.168.56.102:50010 is attempting to report storage ID DS-834248845-192.168.56.102-50010-1415256060466. Node 192.168.56.103:50010 is expected to serve this stor
age.
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDatanode(FSNamesystem.java:4776)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.processReport(FSNamesystem.java:3628)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.blockReport(NameNode.java:1041)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)

        at org.apache.hadoop.ipc.Client.call(Client.java:1107)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
        at $Proxy5.blockReport(Unknown Source)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1026)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
        at java.lang.Thread.run(Thread.java:662)

2014-12-10 17:41:33,330 INFO org.mortbay.log: Stopped SelectChannelConnector@0.0.0.0:50075
2014-12-10 17:41:33,341 INFO org.apache.hadoop.ipc.Server: Stopping server on 50020
2014-12-10 17:41:33,342 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 50020: exiting
2014-12-10 17:41:33,343 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 50020: exiting
2014-12-10 17:41:33,343 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 50020: exiting
2014-12-10 17:41:33,344 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 50020
2014-12-10 17:41:33,346 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2014-12-10 17:41:33,346 INFO org.apache.hadoop.ipc.metrics.RpcInstrumentation: shut down
2014-12-10 17:41:33,347 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(192.168.56.102:50010, storageID=DS-834248845-192.168.56.102-50010-1415256060466, infoPort=
50075, ipcPort=50020):DataXceiveServer:java.nio.channels.AsynchronousCloseException
        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:159)
        at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
        at java.lang.Thread.run(Thread.java:662)

2014-12-10 17:41:33,348 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting DataXceiveServer
2014-12-10 17:41:33,348 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 0
2014-12-10 17:41:33,352 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Exiting DataBlockScanner thread.
2014-12-10 17:41:33,355 INFO org.apache.hadoop.hdfs.server.datanode.FSDatasetAsyncDiskService: Shutting down all async disk service threads...
2014-12-10 17:41:33,355 INFO org.apache.hadoop.hdfs.server.datanode.FSDatasetAsyncDiskService: All async disk service threads have been shut down.
2014-12-10 17:41:33,356 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(192.168.56.102:50010, storageID=DS-834248845-192.168.56.102-50010-1415256060466, infoPort=
50075, ipcPort=50020):Finishing DataNode in: FSDataset{dirpath='/usr/local/hadoop/datanode/current'}
2014-12-10 17:41:33,360 WARN org.apache.hadoop.metrics2.util.MBeans: Hadoop:service=DataNode,name=DataNodeInfo
javax.management.InstanceNotFoundException: Hadoop:service=DataNode,name=DataNodeInfo
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:415)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:403)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:506)
        at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:71)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.unRegisterMXBean(DataNode.java:559)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:805)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1540)
        at java.lang.Thread.run(Thread.java:662)
2014-12-10 17:41:33,362 INFO org.apache.hadoop.ipc.Server: Stopping server on 50020
2014-12-10 17:41:33,362 INFO org.apache.hadoop.ipc.metrics.RpcInstrumentation: shut down
2014-12-10 17:41:33,362 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 0
2014-12-10 17:41:33,362 WARN org.apache.hadoop.metrics2.util.MBeans: Hadoop:service=DataNode,name=FSDatasetState-DS-834248845-192.168.56.102-50010-1415256060466
javax.management.InstanceNotFoundException: Hadoop:service=DataNode,name=FSDatasetState-DS-834248845-192.168.56.102-50010-1415256060466
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:415)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:403)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:506)
        at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:71)
        at org.apache.hadoop.hdfs.server.datanode.FSDataset.shutdown(FSDataset.java:2066)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.shutdown(DataNode.java:867)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1540)
        at java.lang.Thread.run(Thread.java:662)
2014-12-10 17:41:33,363 WARN org.apache.hadoop.hdfs.server.datanode.FSDatasetAsyncDiskService: AsyncDiskService has already shut down.
2014-12-10 17:41:33,363 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2014-12-10 17:41:33,366 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at hadoop2/192.168.56.102
************************************************************/


此异常是因为，两台DataNode的storageID出现了冲突，应该是因为我直接备份安装的原因吧。
解决方法：直接将出现异常的那台slaves机器的data目录删除！


疑点：在我删除slaves异常节点中的data之后，等于是我这台节点上数据已经被删除了，这个会不会对数据有影响，另外疑点就是，我集群是如何数据同步。请大神解答一下，谢谢！！

第一个data目录删除后，你的这个节点数据已经删除掉了。
Hadoop有备份数据，将数据恢复可同步

建议不要hadoop格式化多次

首先你这个版本是多少，感觉很奇怪。
version好像不是hadoop2，但是log由显示的是hadoop2。
data删除自然没有了。如果不是生产环境，可以删除。
但是这并不是唯一的解决方案。
还有log的日期好像也有问题。看看是不是当前的log。不要把日志弄混了。

Joker 发表于 2014-12-13 16:07
建议不要hadoop格式化多次

嗯呢，没有多次格式化hadoop。也不知道是什么原因导致 VERSION的有一个StrogeID  是一样的。所以一台启动成功，另外一台启动就失败了

Joker 发表于 2014-12-13 16:07
第一个data目录删除后，你的这个节点数据已经删除掉了。
Hadoop有备份数据，将数据恢复可同步

嗯呢。不是很明白，hadoop备份数据机制。和怎么做数据恢复的，如果你知道，能不能请帮忙解答一下，谢谢

gefieder 发表于 2014-12-13 16:59
首先你这个版本是多少，感觉很奇怪。
version好像不是hadoop2，但是log由显示的是hadoop2。
data删除自然 ...

我这个版本是hadoop1.1.2的，Version显示的hadoop  是我自己的主机名称。后来考虑到我自己的集群，也没什么数据，就删除了data数据，想请问下，如果我不删除data的话，请教一下其他解决方法？谢谢

韩克拉玛寒 发表于 2014-12-14 14:38
嗯呢。不是很明白，hadoop备份数据机制。和怎么做数据恢复的，如果你知道，能不能请帮忙解答一下，谢谢

集群会自动恢复datanode的数据的。不用手动恢复

学习了~~~~~~~~