scorpionxie 发表于 2016-11-15 15:55:06

hdfs内网集群的访问问题

集群在一个内网环境
机器名内网ip 外网ip部署情况
bigmaster192.168.10.1010.1.1.10 namenode
bignode01192.168.10.11 10.1.1.11 datanode
bignode02192.168.10.12 10.1.1.12 datanode
bignode03192.168.10.13 10.1.1.13 datanode

java程序在10.1.1.55上，已修改10.1.1.55上的hosts，把五个机器名与集群节点的外网IP做了映射。

10.1.1.10 bigmaster
10.1.1.11 bignode01
10.1.1.12 bignode02
10.1.1.13 bignode03

通过java进行文件上传到hdfs的时候，报连接超时。
分析日志看到以下代码：
hdfs.DFSClient: pipeline = DatanodeInfoWithStorage
hdfs.DFSClient: pipeline = DatanodeInfoWithStorage[
hdfs.DFSClient: pipeline = DatanodeInfoWithStorage[

应该是java连接到namenode后，得到的是各个datanode的内网ip，所以无法访问datanode。
求教应该如何解决？

einhep 发表于 2016-11-15 16:12:56

这里面楼主描述的需更加清晰，下面内容
1.集群是如何配置的，用的是内网还是外网ip。
2.开发客户端与集群是什么关系。
3.链接超时，贴出来。
4.这个日志只是显示了一些信息，并非报错。
对于提供的信息，不能确定集群是如何配置的，比如你是统一用的外网配置的集群，还是统一用的内网配置的集群。

scorpionxie 发表于 2016-11-15 16:24:45

einhep 发表于 2016-11-15 16:12
这里面楼主描述的需更加清晰，下面内容
1.集群是如何配置的，用的是内网还是外网ip。
2.开发客户端与集群 ...
1.集群是如何配置的，用的是内网还是外网ip。Q集群是用ambari部署的，是用的机器名。各个节点的hosts文件如下
192.168.10.10 bigmaster
192.168.10.11 bignode01
192.168.10.12 bignode02
192.168.10.13 bignode03

2.开发客户端与集群是什么关系。
Q客户端就是java程序，或者说是开发平台吧，与集群只能通过10.1.1.x进行访问。

3.链接超时，贴出来。

4.这个日志只是显示了一些信息，并非报错。
对于提供的信息，不能确定集群是如何配置的，比如你是统一用的外网配置的集群，还是统一用的内网配置的集群。



日志如下
2016-11-15 15:13:58,816 TRACE ipc.ProtobufRpcEngine: 14: Response <- bigmaster/10.1.1.10:8020: addBlock {block { b { poolId: "BP-1901003557-192.168.1.221-1454661240805" blockId: 1074100545 generationStamp: 359890 numBytes: 0 } offset: 0 locs { id { ipAddr: "192.168.10.13" hostName: "bignode02" datanodeUuid: "f7046005-f515-4b48-b059-88adddb5fd37" xferPort: 50010 infoPort: 50075 ipcPort: 8010 infoSecurePort: 0 } capacity: 1556750635008 dfsUsed: 361574724321 remaining: 1111921903831 blockPoolUsed: 361574724321 lastUpdate: 1479194019243 xceiverCount: 9 location: "/default-rack" adminState: NORMAL cacheCapacity: 0 cacheUsed: 0 lastUpdateMonotonic: 10518160 } locs { id { ipAddr: "192.168.10.11" hostName: "bignode03" datanodeUuid: "f190501c-1309-42f2-89e7-e04b040bc23e" xferPort: 50010 infoPort: 50075 ipcPort: 8010 infoSecurePort: 0 } capacity: 1556750635008 dfsUsed: 361572066017 remaining: 1112808610007 blockPoolUsed: 361572066017 lastUpdate: 1479194019136 xceiverCount: 8 location: "/default-rack" adminState: NORMAL cacheCapacity: 0 cacheUsed: 0 lastUpdateMonotonic: 10518053 } locs { id { ipAddr: "192.168.10.12" hostName: "bignode04" datanodeUuid: "9099adc6-9f68-4a29-ba2a-bb154168c3e4" xferPort: 50010 infoPort: 50075 ipcPort: 8010 infoSecurePort: 0 } capacity: 1556750635008 dfsUsed: 361574056673 remaining: 1112210184407 blockPoolUsed: 361574056673 lastUpdate: 1479194019207 xceiverCount: 8 location: "/default-rack" adminState: NORMAL cacheCapacity: 0 cacheUsed: 0 lastUpdateMonotonic: 10518124 } corrupt: false blockToken { identifier: "\212\001Xh\372\331\r\214\r\275)@\000\000\000\006hadoopuser\000\000\000)BP-1901003557-192.168.1.221-1454661240805\214@\005yA\001\005WRITE" password: "9\335\370D^\222\256\031r\031$\321\353-\370\027Np\356}" kind: "HDFS_BLOCK_TOKEN" service: "" } isCached: false isCached: false isCached: false storageTypes: DISK storageTypes: DISK storageTypes: DISK storageIDs: "DS-fb1f06fa-03f0-4384-ab19-249319fcda06" storageIDs: "DS-a1f3f581-5969-43a3-a133-c038f062300b" storageIDs: "DS-77af5965-1d15-4331-8b53-5099f68a61d7" }}
2016-11-15 15:13:58,819 DEBUG hdfs.DFSClient: pipeline = DatanodeInfoWithStorage
2016-11-15 15:13:58,819 DEBUG hdfs.DFSClient: pipeline = DatanodeInfoWithStorage
2016-11-15 15:13:58,819 DEBUG hdfs.DFSClient: pipeline = DatanodeInfoWithStorage
2016-11-15 15:13:58,819 DEBUG hdfs.DFSClient: Connecting to datanode 192.168.10.13:50010
2016-11-15 15:14:19,817 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.net.ConnectException: Connection timed out: no further information
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
      at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
      at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
      at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1605)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1346)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1299)
      at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464)
2016-11-15 15:14:19,819 INFO hdfs.DFSClient: Abandoning BP-1901003557-192.168.1.221-1454661240805:blk_1074100545_359890
2016-11-15 15:14:19,822 TRACE ipc.ProtobufRpcEngine: 14: Call -> bigmaster/10.1.1.10:8020: abandonBlock {b { poolId: "BP-1901003557-192.168.1.221-1454661240805" blockId: 1074100545 generationStamp: 359890 numBytes: 0 } src: "/apps/bocai/uploadfile/145-500000-0001-20161114170257716/curl.zip" holder: "DFSClient_NONMAPREDUCE_-486330154_1" fileId: 991761}
2016-11-15 15:14:19,823 DEBUG ipc.Client: IPC Client (1327981586) connection to bigmaster/10.1.1.10:8020 from hadoopuser sending #7
2016-11-15 15:14:19,835 DEBUG ipc.Client: IPC Client (1327981586) connection to bigmaster/10.1.1.10:8020 from hadoopuser got value #7
2016-11-15 15:14:19,835 DEBUG ipc.ProtobufRpcEngine: Call: abandonBlock took 14ms
2016-11-15 15:14:19,838 TRACE ipc.ProtobufRpcEngine: 14: Response <- bigmaster/10.1.1.10:8020: abandonBlock {}
2016-11-15 15:14:19,838 INFO hdfs.DFSClient: Excluding datanode DatanodeInfoWithStorage
2016-11-15 15:14:19,844 TRACE ipc.ProtobufRpcEngine: 14: Call -> bigmaster/10.1.1.10:8020: addBlock {src: "/apps/bocai/uploadfile/145-500000-0001-20161114170257716/curl.zip" clientName: "DFSClient_NONMAPREDUCE_-486330154_1" excludeNodes { id { ipAddr: "192.168.10.13" hostName: "bignode02" datanodeUuid: "f7046005-f515-4b48-b059-88adddb5fd37" xferPort: 50010 infoPort: 50075 ipcPort: 8010 infoSecurePort: 0 } capacity: 1556750635008 dfsUsed: 361574724321 remaining: 1111921903831 blockPoolUsed: 361574724321 lastUpdate: 1479194019243 xceiverCount: 9 location: "/default-rack" adminState: NORMAL cacheCapacity: 0 cacheUsed: 0 lastUpdateMonotonic: 10518160 } fileId: 991761}
2016-11-15 15:14:19,845 DEBUG ipc.Client: IPC Client (1327981586) connection to bigmaster/10.1.1.10:8020 from hadoopuser sending #8
2016-11-15 15:14:19,853 DEBUG ipc.Client: IPC Client (1327981586) connection to bigmaster/10.1.1.10:8020 from hadoopuser got value #8
2016-11-15 15:14:19,853 DEBUG ipc.ProtobufRpcEngine: Call: addBlock took 10ms
2016-11-15 15:14:19,855 TRACE ipc.ProtobufRpcEngine: 14: Response <- bigmaster/10.1.1.10:8020: addBlock {block { b { poolId: "BP-1901003557-192.168.1.221-1454661240805" blockId: 1074100546 generationStamp: 359891 numBytes: 0 } offset: 0 locs { id { ipAddr: "192.168.10.12" hostName: "bignode04" datanodeUuid: "9099adc6-9f68-4a29-ba2a-bb154168c3e4" xferPort: 50010 infoPort: 50075 ipcPort: 8010 infoSecurePort: 0 } capacity: 1556750635008 dfsUsed: 361574056673 remaining: 1112210208983 blockPoolUsed: 361574056673 lastUpdate: 1479194040208 xceiverCount: 8 location: "/default-rack" adminState: NORMAL cacheCapacity: 0 cacheUsed: 0 lastUpdateMonotonic: 10539126 } locs { id { ipAddr: "192.168.10.11" hostName: "bignode03" datanodeUuid: "f190501c-1309-42f2-89e7-e04b040bc23e" xferPort: 50010 infoPort: 50075 ipcPort: 8010 infoSecurePort: 0 } capacity: 1556750635008 dfsUsed: 361572066017 remaining: 1112808642775 blockPoolUsed: 361572066017 lastUpdate: 1479194040140 xceiverCount: 8 location: "/default-rack" adminState: NORMAL cacheCapacity: 0 cacheUsed: 0 lastUpdateMonotonic: 10539057 } corrupt: false blockToken { identifier: "\212\001Xh\373+B\214\r\275)@\000\000\000\006hadoopuser\000\000\000)BP-1901003557-192.168.1.221-1454661240805\214@\005yB\001\005WRITE" password: "\336\212\233\221\264\006\245\257\356\232U\266[\206\357A\242\352\313R" kind: "HDFS_BLOCK_TOKEN" service: "" } isCached: false isCached: false storageTypes: DISK storageTypes: DISK storageIDs: "DS-77af5965-1d15-4331-8b53-5099f68a61d7" storageIDs: "DS-a1f3f581-5969-43a3-a133-c038f062300b" }}
2016-11-15 15:14:19,855 DEBUG hdfs.DFSClient: pipeline = DatanodeInfoWithStorage
2016-11-15 15:14:19,855 DEBUG hdfs.DFSClient: pipeline = DatanodeInfoWithStorage
2016-11-15 15:14:19,855 DEBUG hdfs.DFSClient: Connecting to datanode 192.168.10.12:50010
2016-11-15 15:14:28,757 TRACE , created at java.lang.Throwable: TRACE
      at org.apache.hadoop.hdfs.LeaseRenewer.<init>(LeaseRenewer.java:202)
      at org.apache.hadoop.hdfs.LeaseRenewer.<init>(LeaseRenewer.java:71)
      at org.apache.hadoop.hdfs.LeaseRenewer$Factory.get(LeaseRenewer.java:143)
      at org.apache.hadoop.hdfs.LeaseRenewer$Factory.access$100(LeaseRenewer.java:90)
      at org.apache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:80)
      at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:818)
      at org.apache.hadoop.hdfs.DFSClient.beginFileLease(DFSClient.java:824)
      at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1707)
      at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638)
      at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
      at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
      at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444)
      at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
      at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
      at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890)
      at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787)
      at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365)
      at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:356)
      at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
      at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1965)
      at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1933)
      at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1898)
      at com.zd.bocai.hdfs.HdfsUtils.copyFile(HdfsUtils.java:17)
      at com.zd.bocai.BocaiServer.main(BocaiServer.java:24)
] ipc.ProtobufRpcEngine: 15: Call -> bigmaster/10.1.1.10:8020: renewLease {clientName: "DFSClient_NONMAPREDUCE_-486330154_1"}
2016-11-15 15:14:28,758 DEBUG ipc.Client: IPC Client (1327981586) connection to bigmaster/10.1.1.10:8020 from hadoopuser sending #9
2016-11-15 15:14:28,760 DEBUG ipc.Client: IPC Client (1327981586) connection to bigmaster/10.1.1.10:8020 from hadoopuser got value #9
2016-11-15 15:14:28,760 DEBUG , created at java.lang.Throwable: TRACE
      at org.apache.hadoop.hdfs.LeaseRenewer.<init>(LeaseRenewer.java:202)
      at org.apache.hadoop.hdfs.LeaseRenewer.<init>(LeaseRenewer.java:71)
      at org.apache.hadoop.hdfs.LeaseRenewer$Factory.get(LeaseRenewer.java:143)
      at org.apache.hadoop.hdfs.LeaseRenewer$Factory.access$100(LeaseRenewer.java:90)
      at org.apache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:80)
      at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:818)
      at org.apache.hadoop.hdfs.DFSClient.beginFileLease(DFSClient.java:824)
      at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1707)
      at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638)
      at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
      at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
      at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444)
      at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
      at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
      at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890)
      at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787)
      at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365)
      at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:356)
      at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
      at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1965)
      at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1933)
      at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1898)
      at com.zd.bocai.hdfs.HdfsUtils.copyFile(HdfsUtils.java:17)
      at com.zd.bocai.BocaiServer.main(BocaiServer.java:24)
] ipc.ProtobufRpcEngine: Call: renewLease took 3ms

qcbb001 发表于 2016-11-15 20:00:53

scorpionxie 发表于 2016-11-15 16:24
1.集群是如何配置的，用的是内网还是外网ip。Q集群是用ambari部署的，是用的机器名。各个节点的hosts文件 ...
用10段去链接192段肯定不行的。即使bigmaster是双ip也不行啊。两个ip不能通信。
bigmaster/10.1.1.10:8020

需要将下面修改为10段的
192.168.10.10 bigmaster
192.168.10.11 bignode01
192.168.10.12 bignode02
192.168.10.13 bignode03


或则你的开发机器是与192段的同属一个网段才可以
页: [1]
查看完整版本: hdfs内网集群的访问问题