sqoop 1.4.5从mysql导入hdfs出错org.apache.hadoop.yarn.util.Apps.addToEnvironment

本帖最后由 ihadoopi 于 2015-3-3 19:57 编辑

配置：
centos 6.5 64 bits
hadoop 2.4.0
sqoop 1.4.5

都安装好

用如下命令验证sqoop:

..@master-142 ~]$ sqoop list-databases --connect jdbc:mysql://192.168.2.142/ --username root -P

可以正确显示数据列表

但是当尝试从mysql导入数据到hdfs时，

..@master-142 ~]$ sqoop import -m 1 --connect jdbc:mysql://192.168.2.142/logs --username root -P --table weblogs --target-dir /data/weblogs/import

出现如下错误:
15/03/03 12:40:46 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5
15/03/03 12:40:46 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
15/03/03 12:40:47 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
15/03/03 12:40:47 INFO tool.CodeGenTool: Beginning code generation
15/03/03 12:40:48 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `weblogs` AS t LIMIT 1
15/03/03 12:40:48 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `weblogs` AS t LIMIT 1
15/03/03 12:40:48 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop/share/hadoop/mapreduce
Note: /tmp/sqoop-tt/compile/2cbb18a776a1fad22d74868e47c415d5/weblogs.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
15/03/03 12:40:51 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-tt/compile/2cbb18a776a1fad22d74868e47c415d5/weblogs.jar
15/03/03 12:40:51 WARN manager.MySQLManager: It looks like you are importing from mysql.
15/03/03 12:40:51 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
15/03/03 12:40:51 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
15/03/03 12:40:51 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
15/03/03 12:40:51 INFO mapreduce.ImportJobBase: Beginning import of weblogs
15/03/03 12:40:51 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/hbase/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/03/03 12:40:52 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/03/03 12:40:52 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
15/03/03 12:40:53 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
15/03/03 12:40:53 INFO client.RMProxy: Connecting to ResourceManager at master-142/192.168.2.142:18040
15/03/03 12:40:56 INFO db.DBInputFormat: Using read commited transaction isolation
15/03/03 12:40:56 INFO mapreduce.JobSubmitter: number of splits:1
15/03/03 12:40:57 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1425311629716_0015
15/03/03 12:40:57 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/tt/.staging/job_1425311629716_0015
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.yarn.util.Apps.addToEnvironment(Ljava/util/Map;Ljava/lang/String;Ljava/lang/String;)V
at org.apache.hadoop.mapreduce.v2.util.MRApps.setClasspath(MRApps.java:172)
at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:443)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:283)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:430)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286)
at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:247)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:665)
at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

请大家帮忙看看问题

rsgg03 · 发表于 2015-3-3 22:50:05

检查下环境
特别是yarn.application.classpath

 <property>
     <name>yarn.application.classpath</name>
     <value>$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$YARN_HOME/*,$YARN_HOME/lib/*</value>
   </property>
复制代码

ihadoopi · 发表于 2015-3-4 00:14:51

rsgg03 发表于 2015-3-3 22:50
检查下环境
特别是yarn.application.classpath

配置了，问题依旧还在

rsgg03 · 发表于 2015-3-4 00:44:17

ihadoopi 发表于 2015-3-4 00:14
配置了，问题依旧还在

配置的过程中，确保没有空格、换行，路径正确
在配置正确的前提下，确保配置生效。最好重启

ihadoopi · 发表于 2015-3-4 19:45:36

rsgg03 发表于 2015-3-4 00:44
配置的过程中，确保没有空格、换行，路径正确
在配置正确的前提下，确保配置生效。最好重启

你好，按照您说的，我配置重启好，而且把路径在/etc/profile 和 .bashrc都配置了一遍，
#hadoop
export HADOOP_PREFIX=/opt/hadoop
export HADOOP_COMMON_HOME=$HADOOP_PREFIX
export HADOOP_HDFS_HOME=$HADOOP_PREFIX
export HADOOP_MAPRED_HOME=$HADOOP_PREFIX
export YARN_HOME=$HADOOP_PREFIX
export HADOOP_YARN_HOME=$HADOOP_PREFIX
export HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop
export HDFS_CONF_DIR=$HADOOP_PREFIX/etc/hadoop
export YARN_CONF_DIR=$HADOOP_PREFIX/etc/hadoop
export PATH=$PATH:$HADOOP_PREFIX/bin:$HADOOP_PREFIX/sbin:$HADOOP_PREFIX/lib

我还检查了yarn-env.sh,修改了如下:
# resolve links - $0 may be a softlink
(这是之前，发现hadoop2.4没有conf文件)#export YARN_CONF_DIR="${YARN_CONF_DIR:-$HADOOP_YARN_HOME/conf}"
(现在修改后)export YARN_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}

还有我用eclipse写了个简单的java，就是输出System.getenv(),想看看HADOOP_YARN_HOME路径，是对的

以上这些是我所能想到的地方，但是问题依旧是缺那个org.apache.hadoop.yarn.util.Apps.addToEnvironment，用其他pig,hive,hbase都正常。
所以我想请问下，我还可以查看哪些地方，比如用shell命令检查下配置?

xuanxufeng · 发表于 2015-3-4 20:03:36

ihadoopi 发表于 2015-3-4 19:45
你好，按照您说的，我配置重启好，而且把路径在/etc/profile 和 .bashrc都配置了一遍，
#hadoop
export ...

核实下，版本是否兼容。

ihadoopi · 发表于 2015-3-5 02:17:53

xuanxufeng 发表于 2015-3-4 20:03
核实下，版本是否兼容。

终于解决，我试了sqoop1.4.4和1.4.5版本都没有问题

我怎么都没想到原因是在sqoop-env.sh里面路径配置问题:
#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/opt/hadoop/share/hadoop/mapreduce （我跟着一个教程走的）
其实正确的路径应该是：
export HADOOP_MAPRED_HOME=/opt/hadoop

贴上最后的成功运行，以此纪念：

15/03/04 19:09:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/03/04 19:09:22 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
15/03/04 19:09:24 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
15/03/04 19:09:24 INFO client.RMProxy: Connecting to ResourceManager at master-142/192.168.2.142:18040
15/03/04 19:09:27 INFO db.DBInputFormat: Using read commited transaction isolation
15/03/04 19:09:28 INFO mapreduce.JobSubmitter: number of splits:1
15/03/04 19:09:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1425491439164_0005
15/03/04 19:09:29 INFO impl.YarnClientImpl: Submitted application application_1425491439164_0005
15/03/04 19:09:29 INFO mapreduce.Job: The url to track the job: http://master-142:8088/proxy/application_1425491439164_0005/
15/03/04 19:09:29 INFO mapreduce.Job: Running job: job_1425491439164_0005
15/03/04 19:09:45 INFO mapreduce.Job: Job job_1425491439164_0005 running in uber mode : false
15/03/04 19:09:45 INFO mapreduce.Job:  map 0% reduce 0%
15/03/04 19:09:57 INFO mapreduce.Job:  map 100% reduce 0%
15/03/04 19:09:58 INFO mapreduce.Job: Job job_1425491439164_0005 completed successfully
15/03/04 19:09:59 INFO mapreduce.Job: Counters: 30
File System Counters
      FILE: Number of bytes read=0
      FILE: Number of bytes written=104931
      FILE: Number of read operations=0
      FILE: Number of large read operations=0
      FILE: Number of write operations=0
      HDFS: Number of bytes read=87
      HDFS: Number of bytes written=251130
      HDFS: Number of read operations=4
      HDFS: Number of large read operations=0
      HDFS: Number of write operations=2
Job Counters
      Launched map tasks=1
      Other local map tasks=1
      Total time spent by all maps in occupied slots (ms)=10921
      Total time spent by all reduces in occupied slots (ms)=0
      Total time spent by all map tasks (ms)=10921
      Total vcore-seconds taken by all map tasks=10921
      Total megabyte-seconds taken by all map tasks=11183104
Map-Reduce Framework
      Map input records=3000
      Map output records=3000
      Input split bytes=87
      Spilled Records=0
      Failed Shuffles=0
      Merged Map outputs=0
      GC time elapsed (ms)=95
      CPU time spent (ms)=2930
      Physical memory (bytes) snapshot=121692160
      Virtual memory (bytes) snapshot=841236480
      Total committed heap usage (bytes)=15794176
File Input Format Counters
      Bytes Read=0
File Output Format Counters
      Bytes Written=251130
15/03/04 19:09:59 INFO mapreduce.ImportJobBase: Transferred 245.2441 KB in 35.3112 seconds (6.9452 KB/sec)
15/03/04 19:09:59 INFO mapreduce.ImportJobBase: Retrieved 3000 records.

fullmetal · 发表于 2015-7-28 16:20:44

同样的问题解决了，深表感谢啊！！
不得不说，网上的教程只敢对比着看，各种bug！！！

图文精华

sqoop 1.4.5从mysql导入hdfs出错org.apache.hadoop.yarn.util.Apps.addToEnvironment

已有(7)人评论

推荐 /2