图文精华

About云-梭伦科技»专题 › 技术学习(版主发帖区) › 大数据学习 › Sqoop › sqoop导入mysql数据到hive中

sqoop导入mysql数据到hive中

查看数: 67355 | 评论数: 10 | 收藏 0
关灯 | 提示：支持键盘翻页<-左 右->
帖子模式
    组图打开中，请稍候......
howtodown
发布时间: 2014-9-16 15:09

正文摘要:

问题导读： 1.sqoop 在同步mysql表结构到hive找不到包目录的可能原因是什么？ 2.如何查看sqoop同步mysql数据到hive过程？ 3.错误就是hive的jar包有缺失，应该有哪些包？ 一、sqoop 在同步mysql表结 ...

回复

Hentai 发表于 2016-8-4 16:54:04
楼主 我是sqoop2 我想用sqoop连接数据库的时候 一直找不到驱动，我把驱动都放在网上说的那些地方了 这个该怎么解决呢？
credit 发表于 2016-3-17 14:55:13
楼主，做sqoop mysql到hdfs没有问题，但是导入hive报错，这是为什么，
报错：
mapred.jar is deprecated instead use mapreduce.job.jar classnotfound org/kitesdk/data/mapreduce/DatasetKeyOutputFormat
xiaoma 发表于 2015-7-25 20:44:37
楼主我出现问题传不到HDFS

[hadoop@master sqoop-1.4.5]$ bin/sqoop import --connect jdbc:mysql://localhost:3306/sqoop --username root --password root --table tb1 -m 1
15/07/25 20:38:46 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5
15/07/25 20:38:46 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
15/07/25 20:38:47 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
15/07/25 20:38:47 INFO tool.CodeGenTool: Beginning code generation
15/07/25 20:38:47 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tb1` AS t LIMIT 1
15/07/25 20:38:47 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tb1` AS t LIMIT 1
15/07/25 20:38:47 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoop/hadoop-2.2.0
注: /tmp/sqoop-hadoop/compile/91a7d34dd0de631b54bb8f45da12766c/tb1.java使用或覆盖了已过时的 API。
注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。
15/07/25 20:38:50 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/91a7d34dd0de631b54bb8f45da12766c/tb1.jar
15/07/25 20:38:50 WARN manager.MySQLManager: It looks like you are importing from mysql.
15/07/25 20:38:50 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
15/07/25 20:38:50 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
15/07/25 20:38:50 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
15/07/25 20:38:50 INFO mapreduce.ImportJobBase: Beginning import of tb1
15/07/25 20:38:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/07/25 20:38:51 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
15/07/25 20:38:52 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
15/07/25 20:38:52 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.1.20:8032
15/07/25 20:38:58 INFO db.DBInputFormat: Using read commited transaction isolation
15/07/25 20:38:58 INFO mapreduce.JobSubmitter: number of splits:1
15/07/25 20:38:58 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
15/07/25 20:38:58 INFO Configuration.deprecation: mapred.cache.files.timestamps is deprecated. Instead, use mapreduce.job.cache.files.timestamps
15/07/25 20:38:58 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
15/07/25 20:38:58 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
15/07/25 20:38:58 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
15/07/25 20:38:58 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
15/07/25 20:38:58 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
15/07/25 20:38:58 INFO Configuration.deprecation: mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files
15/07/25 20:38:58 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
15/07/25 20:38:58 INFO Configuration.deprecation: mapred.job.classpath.files is deprecated. Instead, use mapreduce.job.classpath.files
15/07/25 20:38:58 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
15/07/25 20:38:58 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
15/07/25 20:38:58 INFO Configuration.deprecation: mapred.cache.files.filesizes is deprecated. Instead, use mapreduce.job.cache.files.filesizes
15/07/25 20:38:58 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
15/07/25 20:38:59 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1437826754583_0003
15/07/25 20:39:00 INFO impl.YarnClientImpl: Submitted application application_1437826754583_0003 to ResourceManager at master/192.168.1.20:8032
15/07/25 20:39:00 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1437826754583_0003/
15/07/25 20:39:00 INFO mapreduce.Job: Running job: job_1437826754583_0003
15/07/25 20:39:24 INFO mapreduce.Job: Job job_1437826754583_0003 running in uber mode : false
15/07/25 20:39:24 INFO mapreduce.Job:  map 0% reduce 0%
15/07/25 20:39:42 INFO mapreduce.Job:  map 100% reduce 0%
15/07/25 20:39:42 INFO mapreduce.Job: Task Id : attempt_1437826754583_0003_m_000000_0, Status : FAILED
Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:725)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
        ... 9 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:408)
        at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1137)
        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:356)
        at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2504)
        at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2541)
        at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2323)
        at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
        at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:408)
        at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
        at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
        ... 10 more
Caused by: java.net.ConnectException: 拒绝连接
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at java.net.Socket.connect(Socket.java:538)
        at java.net.Socket.<init>(Socket.java:434)
        at java.net.Socket.<init>(Socket.java:244)
        at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:258)
        at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:306)
        ... 26 more

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
ainubis 发表于 2015-3-29 04:37:39
O(∩_∩)O谢谢~\(≧▽≦)/~啦啦啦
wbh_qj 发表于 2015-3-27 13:56:58
InvalidObjectException(message:All is already granted by admin)
我也有这些debug信息。貌似没什么影响。


bioger_hit 发表于 2015-2-13 16:04:35
小小布衣 发表于 2015-2-13 15:47
请问解决了吗，我也是同样的问题，
   
      javax.jdo.option.ConnectionURL

如果本地安装，建议安装下面配置

  1. <configuration>  

  2. <property>  

  3.   <name>hive.metastore.warehouse.dir</name>  

  4.   <value>/user/hive_remote/warehouse</value>  

  5. </property>  

  6. 

  7. <property>  

  8.   <name>hive.metastore.local</name>  

  9.   <value>true</value>  

  10. </property>  

  11. 

  12. <property>  

  13.   <name>javax.jdo.option.ConnectionURL</name>  

  14.   <value>jdbc:mysql://localhost/hive_remote?createDatabaseIfNotExist=true</value>  

  15. </property>  

  16. 

  17. <property>  

  18.   <name>javax.jdo.option.ConnectionDriverName</name>  

  19.   <value>com.mysql.jdbc.Driver</value>  

  20. </property>  

  21. 

  22. <property>  

  23.   <name>javax.jdo.option.ConnectionUserName</name>  

  24.   <value>root</value>  

  25. </property>  

  26. 

  27. <property>  

  28.   <name>javax.jdo.option.ConnectionPassword</name>  

  29.   <value>123</value>  

  30. </property>  

  31. </configuration>  
复制代码
更多参考：hbase0.96与hive0.12整合高可靠文档及问题总结






小小布衣 发表于 2015-2-13 15:47:59
请问解决了吗，我也是同样的问题，<configuration>
    <property>
      <name>javax.jdo.option.ConnectionURL</name>
      <value>jdbc:mysql://10.6.0.127:3306/hive?createDatabaseIfNotExist=true</value>
      <description>JDBC connect string for a JDBC metastore</description>
    </property>

    <property>
      <name>javax.jdo.option.ConnectionDriverName</name>
      <value>com.mysql.jdbc.Driver</value>
      <description>Driver class name for a JDBC metastore</description>
    </property>

    <property>
      <name>javax.jdo.option.ConnectionUserName</name>
      <value>hive</value>
      <description>username to use against metastore database</description>
    </property>

    <property>
      <name>javax.jdo.option.ConnectionPassword</name>
      <value>hive</value>
      <description>password to use against metastore database</description>
    </property>
</configuration>
然后我本地使用命令可以直接进入mysql。mysql -h10.6.0.127 -phive -uhive hive  那么这样的话我就是有权限了问题还是这样的debug出现InvalidObjectException(message:Role admin already exists.)
        at org.apache.hadoop.hive.metastore.ObjectStore.addRole(ObjectStore.java:3020)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
wubaozhou 发表于 2015-1-1 20:12:31
howtodown 发表于 2014-9-28 13:01:08
quenlang 发表于 2014-9-28 12:02
楼主，我的调试信息中出现这样的异常，请问是什么原因：
InvalidObjectException(message:Role admin alre ...
应该是元数据库mysql出问题了。可以手工尝试创建hive表
关闭

推荐上一条 /2 下一条

关于我们· 联系我们· 加入我们· 小黑屋· 合作伙伴   

Copyright © 2001-2024 About云-梭伦科技 Powered by Discuz! X3.4 Licensed Discuz Team.

简书 / Smrz 京ICP备2020039040号 Wxb 简书网举报电话：021-34700000