hadoop实战问题总结

本帖最后由 rsgg03 于 2014-1-18 02:25 编辑

1.环境是：jdk和服务器上的jdk都是1.6_45 64位，hive 0.10 并发问题如何解决？
答：首先保证集成编译，如果使用maven最好确定好jdk ，升级到了0.12 并发的问题可解决

2.每条sql逐一测试均正常，就是放到一起产生问题，是什么原因？
答：并发问题

3.出现以下错误：
2014-01-16 18:31:13,014 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.io.IOException: File /input/a.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and no node(s) are excluded in this operation.
2014-01-16 18:31:13,015 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 192.168.1.107:60183 Call#5 Retry#0: error: java.io.IOException: File /input/a.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and no node(s) are excluded in this operation.
java.io.IOException: File /input/a.txt._COPYING_ could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and no node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1384)
答：可以采取以下措施：

删除文件目录
format
启 Namenode, datanode
查看端口 50010, 8020

4.环境是：hadoop 1.0.4 hbase 0.92.1
hbase启动时报错如下：
2014-01-06 13:40:25,372 WARN org.apache.zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:286)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1035)

答：zookeeper默认端口是2181,其他端口2222等产生此错误