分享

Sqoop1.99.5从mysql导入到hdfs出错!Job Failed with status:3

ArsArt 发表于 2015-5-11 09:40:20 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 2 44839
环境是hadoop2.6,集群中有四个节点,运行正常。
Sqoop2  job配置:
[mw_shl_code=text,true]Job with id 1 and name Sqoopy (Enabled: true, Created by root at 5/7/15 5:26 PM, Updated by root at 5/7/15 5:26 PM)
Using link id 2 and Connector id 1
  From database configuration
    Schema name: rural_biz
    Table name: RefundOrder
    Table SQL statement:
    Table column names:
    Partition column name:
    Null value allowed for the partition column:
    Boundary query:
  Throttling resources
    Extractors: 2
    Loaders: 2
  ToJob configuration
    Override null value:
    Null value:
    Output format: TEXT_FILE
    Compression format: NONE
    Custom compression format:
    Output directory: /[/mw_shl_code]
Sqoop2 job运行状态:
[mw_shl_code=text,true]2015-05-08 14:29:12 CST: RUNNING  - 84.47 %
2015-05-08 14:29:22 CST: RUNNING  - 84.47 %
2015-05-08 14:29:32 CST: RUNNING  - 84.47 %
2015-05-08 14:29:42 CST: RUNNING  - 84.47 %
2015-05-08 14:29:52 CST: RUNNING  - 84.47 %
2015-05-08 14:30:02 CST: RUNNING  - 84.47 %
2015-05-08 14:30:13 CST: RUNNING  - 84.47 %
2015-05-08 14:30:23 CST: RUNNING  - 84.47 %
2015-05-08 14:30:33 CST: RUNNING  - 84.47 %
2015-05-08 14:30:43 CST: RUNNING  - 84.47 %
2015-05-08 14:30:53 CST: RUNNING  - 84.47 %
2015-05-08 14:31:04 CST: RUNNING  - 84.47 %
2015-05-08 14:31:14 CST: RUNNING  - 84.47 %
2015-05-08 14:31:24 CST: RUNNING  - 84.47 %
2015-05-08 14:31:34 CST: RUNNING  - 84.47 %
2015-05-08 14:31:44 CST: RUNNING  - 84.47 %
2015-05-08 14:31:54 CST: RUNNING  - 84.47 %
2015-05-08 14:32:04 CST: RUNNING  - 84.47 %
2015-05-08 14:32:14 CST: RUNNING  - 100.00 %
2015-05-08 14:32:47 CST: FAILED
Exception: Job Failed with status:3[/mw_shl_code]
查看Reduce Tack发现reduce一直卡在哪儿但又不会报错。yarn的运行日志没有任何问题。
sqoop.log一直显示
[mw_shl_code=text,true][org.apache.sqoop.security.Authorization.DefaultAuthorizationValidator.checkPrivileges(DefaultAuthorizationValidator.java:39)] privilege: Privilege (Privilege resource: Resource (Resource name: 1, Resource type: JOB ), Privilege action: STATUS, Privilege with_grant_option: false )
2015-05-08 14:49:16,279 INFO  default [org.apache.sqoop.audit.FileAuditLogger.logAuditEvent(FileAuditLogger.java:86)] user=root        ip=127.0.0.1        op=status        obj=job        objId=1
2015-05-08 14:49:16,292 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)] Tx count-begin: 1, rollback: false
2015-05-08 14:49:16,677 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] Tx count-commit: 1, rollback: false
2015-05-08 14:49:16,677 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: 0, rollback: false
2015-05-08 14:49:16,677 INFO  repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)] Attempting transaction commit
2015-05-08 14:51:08,954 DEBUG driver.JobManager [org.apache.sqoop.driver.JobManager$UpdateThread.run(JobManager.java:670)] Updating running submissions
2015-05-08 14:51:08,993 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)] Tx count-begin: 1, rollback: false
2015-05-08 14:51:08,994 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] Tx count-commit: 1, rollback: false
2015-05-08 14:51:08,994 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: 0, rollback: false
2015-05-08 14:51:08,994 INFO  repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)] Attempting transaction commit
2015-05-08 14:56:08,995 DEBUG driver.JobManager [org.apache.sqoop.driver.JobManager$UpdateThread.run(JobManager.java:670)] Updating running submissions
2015-05-08 14:56:09,051 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)] Tx count-begin: 1, rollback: false
2015-05-08 14:56:09,052 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] Tx count-commit: 1, rollback: false
2015-05-08 14:56:09,061 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: 0, rollback: false
2015-05-08 14:56:09,061 INFO  repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)] Attempting transaction commit
2015-05-08 15:01:09,102 DEBUG driver.JobManager [org.apache.sqoop.driver.JobManager$UpdateThread.run(JobManager.java:670)] Updating running submissions
2015-05-08 15:01:09,654 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)] Tx count-begin: 1, rollback: false
2015-05-08 15:01:09,655 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)] Tx count-commit: 1, rollback: false
2015-05-08 15:01:09,655 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)] Tx count-close: 0, rollback: false
2015-05-08 15:01:09,656 INFO  repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)] Attempting transaction commit[/mw_shl_code]
在从节点的job日志中有失败信息显示
[mw_shl_code=text,true]2015-05-08 13:59:29,274 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1430983821763_0004_r_000000_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2015-05-08 13:59:29,274 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1430983821763_0004_r_000001_0: AttemptID:attempt_1430983821763_0004_r_000001_0 Timed out after 600 secs
2015-05-08 13:59:29,275 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1430983821763_0004_r_000001_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
[/code]
[code=text]015-05-08 13:59:29,304 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1430983821763_0004_r_000000_0 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2015-05-08 13:59:29,305 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1430983821763_0004_r_000001_0 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP[/mw_shl_code]

跪求来人帮帮忙阿!

相关帖子

已有(2)人评论

跳转到指定楼层
mituan2008 发表于 2015-5-11 11:30:31
是不是因为数据倾斜的原因,导致变慢了。看楼主有对空值的处理
回复

使用道具 举报

arsenduan 发表于 2015-5-11 12:16:45
日志真看不出来,sqoop语句应该也没有问题,因为正在执行。
所以最有可能的是执行中遇到了不能处理的数据。
按理来说应该有日志,楼主可以细心找找。
贴出来的时间,有些出入。
建议清理日志,重新跑下,这样容易定位错误
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条