从sqoop遇到错误论方法和能力的重要性

本帖最后由 pig2 于 2018-8-19 06:53 编辑

关注最新经典文章，欢迎关注公众号

论方法和能力的重要性：
你是否遇到过这样的情况，当我们编程遇到问题的时候，我们死活搞不定，可是当我们跟我们不太懂这方面技术的领导说明问题后，领导似乎还是不太明白，以为这里完了吗？没有。领导要求你跟他说明详细情况包括涉及到的技术，当领导明白之后，它就可以指导你解决这个问题。
上面是我们的人遇到的亲身经历。这里说下个人的亲身的例子。
对于sqoop，我们是经常使用的。我们的QQ群成员遇到了下面问题：

18/08/17 15:51:16 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0
18/08/17 15:51:17 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
18/08/17 15:51:17 INFO manager.SqlManager: Using default fetchSize of 1000
18/08/17 15:51:17 INFO tool.CodeGenTool: Beginning code generation
18/08/17 15:51:17 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM user_web AS t WHERE 1=0
18/08/17 15:51:17 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM user_web AS t WHERE 1=0
18/08/17 15:51:17 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoop/hadoop
Note: /tmp/sqoop-hadoop/compile/c7d5f23eb822dfb826585804908e38ee/user_web.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
18/08/17 15:51:19 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/c7d5f23eb822dfb826585804908e38ee/user_web.jar
18/08/17 15:51:19 INFO mapreduce.ImportJobBase: Beginning import of user_web
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See 链接：http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/08/17 15:51:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/08/17 15:51:19 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
18/08/17 15:51:19 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM user_web AS t WHERE 1=0
18/08/17 15:51:20 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
18/08/17 15:51:20 INFO client.RMProxy: Connecting to ResourceManager at master/172.28.1.61:8032
18/08/17 15:51:31 INFO db.DBInputFormat: Using read commited transaction isolation
18/08/17 15:51:31 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(createdt), MAX(createdt) FROM user_web
18/08/17 15:51:31 INFO db.IntegerSplitter: Split size: 49593600000; Num splits: 4 from: 1316448000000 to: 1514822400000
18/08/17 15:51:31 INFO mapreduce.JobSubmitter: number of splits:4
18/08/17 15:51:31 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1534401266670_0004
18/08/17 15:51:32 INFO impl.YarnClientImpl: Submitted application application_1534401266670_0004
18/08/17 15:51:32 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1534401266670_0004/
18/08/17 15:51:32 INFO mapreduce.Job: Running job: job_1534401266670_0004
18/08/17 15:51:40 INFO mapreduce.Job: Job job_1534401266670_0004 running in uber mode : false
18/08/17 15:51:40 INFO mapreduce.Job: map 0% reduce 0%
18/08/17 15:51:45 INFO mapreduce.Job: Task Id : attempt_1534401266670_0004_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Unable to load locale categories.
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:746)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)

QQ图片20180819062829.jpg