sqoop export 报错

[hadoop@master ~]$ sqoop-export --connect jdbc:mysql://localhost:3306/te_hive --username sqoop --password sqoop --table tb_test_insertFile --export-dir /user/hive_remote/warehouse/tb_test_insertfile/input.txt
Warning: /opt/sqoop/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/sqoop/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
17/02/06 20:25:42 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
17/02/06 20:25:42 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
17/02/06 20:25:42 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
17/02/06 20:25:42 INFO tool.CodeGenTool: Beginning code generation
17/02/06 20:25:42 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tb_test_insertFile` AS t LIMIT 1
17/02/06 20:25:42 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tb_test_insertFile` AS t LIMIT 1
17/02/06 20:25:42 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop/hadoop-2.7.0
Note: /tmp/sqoop-hadoop/compile/4a314019c7f2a6cbed21656d8aa418a2/tb_test_insertFile.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/02/06 20:25:44 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/4a314019c7f2a6cbed21656d8aa418a2/tb_test_insertFile.jar
17/02/06 20:25:44 INFO mapreduce.ExportJobBase: Beginning export of tb_test_insertFile
17/02/06 20:25:44 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hadoop/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/hbase-1.2.3/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
17/02/06 20:25:45 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
17/02/06 20:25:45 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.UnsupportedOperationException: Not implemented by the S3FileSystem FileSystem implementation
java.lang.UnsupportedOperationException: Not implemented by the S3FileSystem FileSystem implementation
at org.apache.hadoop.fs.FileSystem.getScheme(FileSystem.java:217)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2624)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2634)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
at org.apache.sqoop.mapreduce.ExportJobBase.getInputPath(ExportJobBase.java:223)
at org.apache.sqoop.mapreduce.ExportJobBase.getInputFileType(ExportJobBase.java:505)
at org.apache.sqoop.mapreduce.JdbcExportJob.configureInputFormat(JdbcExportJob.java:68)
at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:424)
at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:912)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:81)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
[hadoop@master ~]$

yuwenge · 发表于 2017-2-7 08:27:17

可能是配置的问题，s3是亚马逊存储系统。

Wyy_Ck · 发表于 2017-2-7 09:16:07

yuwenge 发表于 2017-2-7 08:27
可能是配置的问题，s3是亚马逊存储系统。

我的hadoop是原生态的hadoop， sqoop难道需要和CDH hadoop 般配吗

qcbb001 · 发表于 2017-2-7 09:42:01

本帖最后由 qcbb001 于 2017-2-7 09:44 编辑

这里面存在很多的问题，首先最基本的sql语句应该就有问题
sqoop-export --connect jdbc:mysql://localhost:3306/te_hive --username sqoop --password sqoop --table tb_test_insertFile --export-dir /user/hive_remote/warehouse/tb_test_insertfile/input.txt
红字部分应该是目录，也就是一个表，而不是一个文件
还有上面要弄清楚export 与import作用

另外还需要注意sqoop与hadoop版本的兼容性

Wyy_Ck · 发表于 2017-2-7 10:06:53

qcbb001 发表于 2017-2-7 09:42
这里面存在很多的问题，首先最基本的sql语句应该就有问题
sqoop-export --connect jdbc:mysql://localhos ...

非常感谢您。我感觉我这边是不是有问题哈。
1、我通过Load Data命令将数据上传到目录，如下：
[mw_shl_code=shell,true][hadoop@master ~]$ hadoop fs -cat /user/hive_remote/warehouse/tb_test_insertfile/input.txt
1|name1|24
2|name2|20
3|name3|40[/mw_shl_code]

在hive中查询如下：
[mw_shl_code=shell,true]hive> create table tb_test_insertFile (id int ,name String, age int) row format delimited fields terminated by '|' stored as TEXTFILE;
OK
Time taken: 1.416 seconds
hive> load data local inpath '/opt/hive/test/input.txt' into tb_test_insertFile;
FAILED: ParseException line 1:55 missing TABLE at 'tb_test_insertFile' near '<EOF>'
hive> load data local inpath '/opt/hive/test/input.txt' into table tb_test_insertFile;
Loading data to table default.tb_test_insertfile
OK
Time taken: 0.664 seconds
hive> select * from tb_test_insertFile;
OK
1 name1 24
2 name2 20
3 name3 40
Time taken: 1.114 seconds, Fetched: 3 row(s)
hive> [/mw_shl_code]

我现在想把通过mysql客户端查看表的数据导入到mysql客户端，不知道哪里出错了或许我糊涂了？

O(∩_∩)O谢谢