各位好,这两天部署了一套3节点的CDH集群做测试,尝试使用Sqoop2从Mysql向Hbase中导入数据,过程中遇到了报错,尝试解决未果,请各位帮忙看一下是否有解决思路。
环境:
Centos 6.5 64bit
Cloudera Manager 5.4
CDH 5.3.3
CDH中的相关组件版本为:
Sqoop2 1.99.4
hbase 0.98.6
hadoop 2.5.0
Hbase的JDBC Driver使用的是phoenix-4.3.1。
现在create link & job 都已经通过了,但是start job时报错:
java.lang.IllegalAccessError: class com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass com.google.protobuf.LiteralByteString
/opt/cloudera/parcels/CDH/lib/hadoop/ /opt/cloudera/parcels/CDH/lib/hadoop/lib
下面是供参考的建立link & job的过程和完整报错:
建立Link(下面是建立hbase link的过程,建立mysql的link过程略): sqoop:000> create link -c 2 Creating link for connector with id 2 Please fill following values to create new link object Name: hbasetest
Link configuration
JDBC Driver Class: org.apache.phoenix.jdbc.PhoenixDriver JDBC Connection String: jdbc:phoenix:master:2181 Username: Password: JDBC Connection Properties: There are currently 0 values in the map: entry# New link was successfully created with validation status OK and persistent id 14
建立job: sqoop:000> create job -f 6 -t 14 Creating job for links with from id 6 and to id 14 Please fill following values to create new job object Name: MysqlToHbase
From database configuration
Schema name: clouderam Table name: users Table SQL statement: Table column names: Partition column name: Null value allowed for the partition column: Boundary query:
To database configuration
Schema name: main Table name: users Table SQL statement: Table column names: Stage table name: Should clear stage table:
Throttling resources
Extractors: Loaders:
New job was successfully created with validation status OK and persistent id 4
完整报错,日志文件为:/var/log/sqoop2/sqoop-cmf-sqoop-SQOOP_SERVER-slave2.log.out 2015-05-12 17:23:26,138 INFO org.apache.sqoop.repository.JdbcRepositoryTransaction: Attempting transaction commit 2015-05-12 17:23:26,417 INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer: Boundaries: min=1, max=4, columnType=-5 2015-05-12 17:23:26,419 INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer: Using dataSql: SELECT * FROM clouderam.users WHERE ${CONDITIONS} 2015-05-12 17:23:26,419 INFO org.apache.sqoop.connector.jdbc.GenericJdbcFromInitializer: Field names: USER_ID,USER_NAME,PASSWORD_HASH,PASSWORD_SALT,PASSWORD_LOGIN,OPTIMISTIC_LOCK_VERSION 2015-05-12 17:23:29,446 ERROR org.apache.sqoop.connector.jdbc.GenericJdbcExecutor: Caught SQLException: java.sql.SQLException: java.lang.IllegalAccessError: class com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass com.google.protobuf.LiteralByteString at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1024) at org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1257) at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:350) at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:311) at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:307) at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:333) at org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:237) at org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:160) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:340) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:330) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:240) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:235) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:234) at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:1073) at org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.getQueryColumns(GenericJdbcExecutor.java:227) at org.apache.sqoop.connector.jdbc.GenericJdbcToInitializer.configureTableProperties(GenericJdbcToInitializer.java:169) at org.apache.sqoop.connector.jdbc.GenericJdbcToInitializer.initialize(GenericJdbcToInitializer.java:49) at org.apache.sqoop.connector.jdbc.GenericJdbcToInitializer.initialize(GenericJdbcToInitializer.java:39) at org.apache.sqoop.driver.JobManager.initializeConnector(JobManager.java:451) at org.apache.sqoop.driver.JobManager.createJobRequest(JobManager.java:372) at org.apache.sqoop.driver.JobManager.start(JobManager.java:277) at org.apache.sqoop.handler.JobRequestHandler.startJob(JobRequestHandler.java:367) at org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:112) at org.apache.sqoop.server.v1.JobServlet.handlePutRequest(JobServlet.java:96) at org.apache.sqoop.server.SqoopProtocolServlet.doPut(SqoopProtocolServlet.java:79) at javax.servlet.http.HttpServlet.service(HttpServlet.java:646) at javax.servlet.http.HttpServlet.service(HttpServlet.java:723) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:555) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:620) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalAccessError: class com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass com.google.protobuf.LiteralByteString at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2854) at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1159) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1647) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526) at org.apache.phoenix.query.ConnectionQueryServicesImpl$7.call(ConnectionQueryServicesImpl.java:1265) at org.apache.phoenix.query.ConnectionQueryServicesImpl$7.call(ConnectionQueryServicesImpl.java:1258) at org.apache.hadoop.hbase.client.HTable$17.call(HTable.java:1571) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ... 1 more 2015-05-12 17:23:29,463 ERROR org.apache.sqoop.server.SqoopProtocolServlet: Exception in PUT http://slave2:15002/sqoop/v1/job/4/start org.apache.sqoop.common.SqoopException: GENERIC_JDBC_CONNECTOR_0003:Unable to access meta data at org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.getQueryColumns(GenericJdbcExecutor.java:240) at org.apache.sqoop.connector.jdbc.GenericJdbcToInitializer.configureTableProperties(GenericJdbcToInitializer.java:169) at org.apache.sqoop.connector.jdbc.GenericJdbcToInitializer.initialize(GenericJdbcToInitializer.java:49) at org.apache.sqoop.connector.jdbc.GenericJdbcToInitializer.initialize(GenericJdbcToInitializer.java:39) at org.apache.sqoop.driver.JobManager.initializeConnector(JobManager.java:451) at org.apache.sqoop.driver.JobManager.createJobRequest(JobManager.java:372) at org.apache.sqoop.driver.JobManager.start(JobManager.java:277) at org.apache.sqoop.handler.JobRequestHandler.startJob(JobRequestHandler.java:367) at org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:112) at org.apache.sqoop.server.v1.JobServlet.handlePutRequest(JobServlet.java:96) at org.apache.sqoop.server.SqoopProtocolServlet.doPut(SqoopProtocolServlet.java:79) at javax.servlet.http.HttpServlet.service(HttpServlet.java:646) at javax.servlet.http.HttpServlet.service(HttpServlet.java:723) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:555) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:620) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:745) Caused by: java.sql.SQLException: java.lang.IllegalAccessError: class com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass com.google.protobuf.LiteralByteString at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1024) at org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1257) at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:350) at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:311) at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:307) at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:333) at org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:237) at org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:160) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:340) at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:330) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:240) at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:235) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:234) at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:1073) at org.apache.sqoop.connector.jdbc.GenericJdbcExecutor.getQueryColumns(GenericJdbcExecutor.java:227) ... 28 more Caused by: java.lang.IllegalAccessError: class com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass com.google.protobuf.LiteralByteString at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:2854) at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:1159) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1647) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1526) at org.apache.phoenix.query.ConnectionQueryServicesImpl$7.call(ConnectionQueryServicesImpl.java:1265) at org.apache.phoenix.query.ConnectionQueryServicesImpl$7.call(ConnectionQueryServicesImpl.java:1258) at org.apache.hadoop.hbase.client.HTable$17.call(HTable.java:1571) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ... 1 more 2015-05-12 17:25:51,148 INFO org.apache.sqoop.repository.JdbcRepositoryTransaction: Attempting transaction commit
2015-05-12 17:30:51,149 INFO org.apache.sqoop.repository.JdbcRepositoryTransaction: Attempting transaction commit
|