分享

Sqoop将Hive表中数据导入MySQL中,进程死住,不执行该怎么办

NIITYZU 发表于 2015-4-21 20:05:15 [显示全部楼层] 只看大图 回帖奖励 阅读模式 关闭右栏 3 31990
我在使用Sqoop将Hive分析结果数据导入到MySQL中时候,第一次执行成功了,再次执行的时候,进程死住了,一直保持如下状态,不再往下执行,如下
  1. [hadoopUser@secondmgt ~]$ sqoop-export --connect jdbc:mysql://secondmgt:3306/transport --username hive --password hive --table jtxx --export-dir /hive/warehouse/hive.db/temptrans --input-fields-terminated-by '\001'
  2. Warning: /usr/lib/hcatalog does not exist! HCatalog jobs will fail.
  3. Please set $HCAT_HOME to the root of your HCatalog installation.
  4. 15/04/21 20:08:28 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
  5. 15/04/21 20:08:28 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
  6. 15/04/21 20:08:28 INFO tool.CodeGenTool: Beginning code generation
  7. 15/04/21 20:08:29 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `jtxx` AS t LIMIT 1
  8. 15/04/21 20:08:29 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `jtxx` AS t LIMIT 1
  9. 15/04/21 20:08:29 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoopUser/cloud/hadoop/programs/hadoop-2.2.0
  10. Note: /tmp/sqoop-hadoopUser/compile/67173774b957b511b4d62bc4ebe56e23/jtxx.java uses or overrides a deprecated API.
  11. Note: Recompile with -Xlint:deprecation for details.
  12. 15/04/21 20:08:30 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoopUser/compile/67173774b957b511b4d62bc4ebe56e23/jtxx.jar
  13. 15/04/21 20:08:30 INFO mapreduce.ExportJobBase: Beginning export of jtxx
  14. 15/04/21 20:08:30 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
  15. SLF4J: Class path contains multiple SLF4J bindings.
  16. SLF4J: Found binding in [jar:file:/home/hadoopUser/cloud/hadoop/programs/hadoop-2.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  17. SLF4J: Found binding in [jar:file:/home/hadoopUser/cloud/hbase/hbase-0.96.2-hadoop2/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  18. SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  19. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  20. 15/04/21 20:08:30 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
  21. 15/04/21 20:08:31 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
  22. 15/04/21 20:08:31 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
  23. 15/04/21 20:08:31 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
  24. 15/04/21 20:08:31 INFO client.RMProxy: Connecting to ResourceManager at secondmgt/192.168.2.133:8032
  25. 15/04/21 20:08:32 INFO input.FileInputFormat: Total input paths to process : 2
  26. 15/04/21 20:08:32 INFO input.FileInputFormat: Total input paths to process : 2
  27. 15/04/21 20:08:32 INFO mapreduce.JobSubmitter: number of splits:3
  28. 15/04/21 20:08:32 INFO Configuration.deprecation: mapred.job.classpath.files is deprecated. Instead, use mapreduce.job.classpath.files
  29. 15/04/21 20:08:32 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
  30. 15/04/21 20:08:32 INFO Configuration.deprecation: mapred.cache.files.filesizes is deprecated. Instead, use mapreduce.job.cache.files.filesizes
  31. 15/04/21 20:08:32 INFO Configuration.deprecation: mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files
  32. 15/04/21 20:08:32 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
  33. 15/04/21 20:08:32 INFO Configuration.deprecation: mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class
  34. 15/04/21 20:08:32 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
  35. 15/04/21 20:08:32 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
  36. 15/04/21 20:08:32 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
  37. 15/04/21 20:08:32 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
  38. 15/04/21 20:08:32 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
  39. 15/04/21 20:08:32 INFO Configuration.deprecation: mapred.cache.files.timestamps is deprecated. Instead, use mapreduce.job.cache.files.timestamps
  40. 15/04/21 20:08:32 INFO Configuration.deprecation: mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class
  41. 15/04/21 20:08:32 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
  42. 15/04/21 20:08:32 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1428394594787_0016
  43. 15/04/21 20:08:33 INFO impl.YarnClientImpl: Submitted application application_1428394594787_0016 to ResourceManager at secondmgt/192.168.2.133:8032
  44. 15/04/21 20:08:33 INFO mapreduce.Job: The url to track the job: http://secondmgt:8088/proxy/application_1428394594787_0016/
  45. 15/04/21 20:08:33 INFO mapreduce.Job: Running job: job_1428394594787_0016
复制代码
我查看了一下,发现是应用程序队列被100%占用,此时该怎么释放啊?请大神指点,谢谢
01.png

已有(3)人评论

跳转到指定楼层
desehawk 发表于 2015-4-21 22:13:04
最坏的方法是把进程杀死
回复

使用道具 举报

evababy 发表于 2015-5-11 10:41:03
我也碰到同样问题,测试场景:
MYSQL有主键约束,而HIVE数据重复,HIVE到MYSQL过程中出现僵死(未查看详细日志,必然报SQL错误),最后手动杀掉执行进程的。
此现象直接导致程序无法结束(也会因为可能存在异常而去实时检测LOG),如果在执行过程中产生异常时SQOOP能自动退出、或获取错误堆栈就好了,可惜啊
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条