HA+Federation中datanode格式化出现异常...不能为多个namenode提供服务

求大神...自己一个人尝试了两天了,一直是这个报错,如下
2016-05-20 17:07:48,804 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to hadoop1/192.168.2.10:9000. Exiting.
java.io.IOException: Cluster IDs not matched: dn cid=cluster1 but ns cid=CID-91fad4b9-26cf-4c08-8466-1035509f4880; bpid=BP-841114638-192.168.2.10-1463734832788
         at org.apache.hadoop.hdfs.server.datanode.DataNode.setClusterId(DataNode.java:717)
         at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1316)
         at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:317)
         at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
         at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:802)
         at java.lang.Thread.run(Thread.java:745)
2016-05-20 17:07:48,804 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to hadoop2/192.168.2.11:9000. Exiting.
java.io.IOException: Cluster IDs not matched: dn cid=cluster1 but ns cid=CID-91fad4b9-26cf-4c08-8466-1035509f4880; bpid=BP-841114638-192.168.2.10-1463734832788
         at org.apache.hadoop.hdfs.server.datanode.DataNode.setClusterId(DataNode.java:717)
         at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1316)
         at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:317)
         at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:223)
         at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:802)
         at java.lang.Thread.run(Thread.java:745)

gao634209276 · 发表于 2016-5-20 22:24:31

这是第一次启动datanode时格式化时候,抛的异样.只能将一个namode的cid给格式化过来,另一个不能,然后虽然所有的datanode都启动了,但是在两个Active  NameNode中使用web UI查看live的datanode或者在cli中使用dfsadmin -report的时候,datanode的节点只会显示在格式化时候使用的cid对应的namode下面.例如下面在连个不同的ns下的namnode中使用-report的打印的结果:

[hadoop@slave1 hadoop-2.7.2]$ bin/hdfs dfsadmin -report
16/05/20 17:17:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Configured Capacity: 130101776384 (121.17 GB)
Present Capacity: 92205232128 (85.87 GB)
DFS Remaining: 92205158400 (85.87 GB)
DFS Used: 73728 (72 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (3):

Name: 192.168.2.3:50010 (hadoop)
Hostname: hadoop
Decommission Status : Normal
Configured Capacity: 97124175872 (90.45 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 25148153856 (23.42 GB)
DFS Remaining: 71975997440 (67.03 GB)
DFS Used%: 0.00%
DFS Remaining%: 74.11%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri May 20 17:17:07 CST 2016


Name: 192.168.2.8:50010 (slave2)
Hostname: slave2
Decommission Status : Normal
Configured Capacity: 16488800256 (15.36 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 5681549312 (5.29 GB)
DFS Remaining: 10807226368 (10.07 GB)
DFS Used%: 0.00%
DFS Remaining%: 65.54%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri May 20 17:17:06 CST 2016


Name: 192.168.2.7:50010 (slave1)
Hostname: slave1
Decommission Status : Normal
Configured Capacity: 16488800256 (15.36 GB)
DFS Used: 24576 (24 KB)
Non DFS Used: 7066877952 (6.58 GB)
DFS Remaining: 9421897728 (8.77 GB)
DFS Used%: 0.00%
DFS Remaining%: 57.14%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri May 20 17:17:09 CST 2016

[hadoop@hadoop1 hadoop-2.7.2]$ bin/hdfs dfsadmin -report
16/05/20 17:59:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Configured Capacity: 32977600512 (30.71 GB)
Present Capacity: 24352546816 (22.68 GB)
DFS Remaining: 24352387072 (22.68 GB)
DFS Used: 159744 (156 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0

-------------------------------------------------
Live datanodes (2):

Name: 192.168.2.11:50010 (hadoop2)
Hostname: hadoop2
Decommission Status : Normal
Configured Capacity: 16488800256 (15.36 GB)
DFS Used: 77824 (76 KB)
Non DFS Used: 4311064576 (4.01 GB)
DFS Remaining: 12177657856 (11.34 GB)
DFS Used%: 0.00%
DFS Remaining%: 73.85%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri May 20 17:59:29 CST 2016


Name: 192.168.2.10:50010 (hadoop1)
Hostname: hadoop1
Decommission Status : Normal
Configured Capacity: 16488800256 (15.36 GB)
DFS Used: 81920 (80 KB)
Non DFS Used: 4313989120 (4.02 GB)
DFS Remaining: 12174729216 (11.34 GB)
DFS Used%: 0.00%
DFS Remaining%: 73.84%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Fri May 20 17:59:29 CST 2016

gao634209276 · 发表于 2016-5-20 22:32:37

每个namenode随机独占n个datanode,也就是说datanode不能提供给两个namenode,比如说在hadoop1节点使用cli进行put文件,只能put到hadoop1和hadoop2的datanode中,永远不能put到slave1,slave2和hadoop的datanode中...当时我专门测试上传文件查看块信息...最后我不信,只设置一个datanode,然后依格式化启动,两对namenode就有一对namenode能占有这个datanode

langke93 · 发表于 2016-5-21 06:55:59

gao634209276 发表于 2016-5-20 22:32
每个namenode随机独占n个datanode,也就是说datanode不能提供给两个namenode,比如说在hadoop1节点使用cli进 ...

Cluster IDs not matched: dn cid=cluster1 but ns cid=CID-91fad4b9-26cf-4c08-8466-1035509f4880; bpid=BP-841114638-192.168.2.10-1463734832788
让他们的id保持一致

gao634209276 · 发表于 2016-5-22 20:18:11

langke93 发表于 2016-5-21 06:55
Cluster IDs not matched: dn cid=cluster1 but ns cid=CID-91fad4b9-26cf-4c08-8466-1035509f4880; bpi ...

因为我这个集群使用了联邦,有两个nameService,这个cid=cluster1是其中一个的,bpid也是这个ns下对应的存储池的id,按照联邦的理论,应该是每个datanode上会在初始化时候就把所有的ns下的cid和bpid均保存下来,提供给所有ns存储,读写的服务,但是我这里为何总是只能保存其中一个...每个datanode只能被其中一个ns管理!!!
复制也不是事啊,关键是第一启动就出现着毛病,不知道哪里的事情,有没有和我一样的情况,毕竟我看过好多Ha联邦的配置,在格式化之前也都把以前的存储文件删除完了,重新配置和格式化启动都尝试了估计有十来次,都是着毛病...