图文精华

About云-梭伦科技»专题 › 交流区 › 技术交流 › HDFS › 【求助】hadoop0.20.1如何配置lzo压缩？
分享

【求助】hadoop0.20.1如何配置lzo压缩？

a2325145 发表于 2013-10-25 10:46:11 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 7 9055
各位好，我的hadoop是0.20.1，现在想使用 lzo压缩，目前在编译hadoop-gpl-compression的时候出错，错误信息如下：
[ol]
  • [root@linux-1 hadoop-gpl-compression-0.1.0]# export CFLAGS=-m32
  • [root@linux-1 hadoop-gpl-compression-0.1.0]# export CXXFLAGS=-m32
  • [root@linux-1 hadoop-gpl-compression-0.1.0]# ant compile-native
  • Buildfile: /soft/hadoop-gpl-compression-0.1.0/build.xml
  • ivy-download:
  •       [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.0.0-rc2/ivy-2.0.0-rc2.jar
  •       [get] To: /soft/hadoop-gpl-compression-0.1.0/ivy/ivy-2.0.0-rc2.jar
  •       [get] Not modified - so not downloaded
  • ivy-init-dirs:
  • ivy-probe-antlib:
  • ivy-init-antlib:
  • ivy-init:
  • [ivy:configure] :: Ivy 2.0.0-rc2 - 20081028224207 :: http://ant.apache.org/ivy/ ::
  • :: loading settings :: file = /soft/hadoop-gpl-compression-0.1.0/ivy/ivysettings.xml
  • ivy-resolve-common:
  • [ivy:resolve] :: resolving dependencies :: com.hadoop.gplcompression#Hadoop-GPL-Compression;working@linux-1
  • [ivy:resolve]   confs: [common]
  • [ivy:resolve]   found commons-logging#commons-logging;1.0.4 in maven2
  • [ivy:resolve]   found junit#junit;3.8.1 in maven2
  • [ivy:resolve]   found commons-logging#commons-logging-api;1.0.4 in maven2
  • [ivy:resolve] :: resolution report :: resolve 284ms :: artifacts dl 4ms
  •         ---------------------------------------------------------------------
  •         |                  |            modules            ||   artifacts   |
  •         |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
  •         ---------------------------------------------------------------------
  •         |      common      |   3   |   0   |   0   |   0   ||   3   |   0   |
  •         ---------------------------------------------------------------------
  • ivy-retrieve-common:
  • [ivy:retrieve] :: retrieving :: com.hadoop.gplcompression#Hadoop-GPL-Compression
  • [ivy:retrieve]  confs: [common]
  • [ivy:retrieve]  0 artifacts copied, 3 already retrieved (0kB/12ms)
  • No ivy:settings found for the default reference 'ivy.instance'.  A default instance will be used
  • DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead
  • :: loading settings :: file = /soft/hadoop-gpl-compression-0.1.0/ivy/ivysettings.xml
  • init:
  • compile-java:
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/build.xml:191: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
  •     [javac] Compiling 9 source files to /soft/hadoop-gpl-compression-0.1.0/build/classes
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/compression/lzo/LzoCodec.java:27: package org.apache.hadoop.conf does not exist
  •     [javac] import org.apache.hadoop.conf.Configurable;
  •     [javac]                              ^
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/compression/lzo/LzoCodec.java:28: package org.apache.hadoop.conf does not exist
  •     [javac] import org.apache.hadoop.conf.Configuration;
  •     [javac]                              ^
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/compression/lzo/LzoCodec.java:29: cannot find symbol
  •     [javac] symbol  : class BlockCompressorStream
  •     [javac] location: package org.apache.hadoop.io.compress
  •     [javac] import org.apache.hadoop.io.compress.BlockCompressorStream;
  •     [javac]                                     ^
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/compression/lzo/LzoCodec.java:30: cannot find symbol
  •     [javac] symbol  : class BlockDecompressorStream
  •     [javac] location: package org.apache.hadoop.io.compress
  •     [javac] import org.apache.hadoop.io.compress.BlockDecompressorStream;
  •     [javac]                                     ^
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/compression/lzo/LzoCodec.java:31: cannot find symbol
  •     [javac] symbol  : class CompressionCodec
  •     [javac] location: package org.apache.hadoop.io.compress
  •     [javac] import org.apache.hadoop.io.compress.CompressionCodec;
  •     [javac]                                     ^
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/compression/lzo/LzoCodec.java:32: cannot find symbol
  •     [javac] symbol  : class CompressionInputStream
  •     [javac] location: package org.apache.hadoop.io.compress
  •     [javac] import org.apache.hadoop.io.compress.CompressionInputStream;
  •     [javac]                                     ^
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/compression/lzo/LzoCodec.java:33: cannot find symbol
  •     [javac] symbol  : class CompressionOutputStream
  •     [javac] location: package org.apache.hadoop.io.compress
  •     [javac] import org.apache.hadoop.io.compress.CompressionOutputStream;
  •     [javac]                                     ^
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/compression/lzo/LzoCodec.java:34: cannot find symbol
  •     [javac] symbol  : class Compressor
  •     [javac] location: package org.apache.hadoop.io.compress
  •     [javac] import org.apache.hadoop.io.compress.Compressor;
  •     [javac]                                     ^
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/compression/lzo/LzoCodec.java:35: cannot find symbol
  •     [javac] symbol  : class Decompressor
  •     [javac] location: package org.apache.hadoop.io.compress
  •     [javac] import org.apache.hadoop.io.compress.Decompressor;
  •     [javac]                                     ^
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/compression/lzo/LzoCodec.java:43: cannot find symbol
  •     [javac] symbol: class Configurable
  •     [javac] public class LzoCodec implements Configurable, CompressionCodec {
  •     [javac]                                  ^
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/compression/lzo/LzoCodec.java:43: cannot find symbol
  •     [javac] symbol: class CompressionCodec
  •     [javac] public class LzoCodec implements Configurable, CompressionCodec {
  •     [javac]                                                ^
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/compression/lzo/LzoCodec.java:47: cannot find symbol
  •     [javac] symbol  : class Configuration
  •     [javac] location: class com.hadoop.compression.lzo.LzoCodec
  •     [javac]   private Configuration conf;
  •     [javac]           ^
  •     [javac]   public Decompressor createDecompressor() {
  •     [javac]          ^
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/compression/lzo/LzopCodec.java:135: cannot find symbol
  •     [javac] symbol  : class BlockCompressorStream
  •     [javac] location: class com.hadoop.compression.lzo.LzopCodec
  •     [javac]   protected static class LzopOutputStream extends BlockCompressorStream {
  •     [javac]                                                   ^
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/compression/lzo/LzopCodec.java:179: cannot find symbol
  •     [javac] symbol  : class Compressor
  •     [javac] location: class com.hadoop.compression.lzo.LzopCodec.LzopOutputStream
  •     [javac]     public LzopOutputStream(OutputStream out, Compressor compressor,
  •     [javac]                                               ^
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/compression/lzo/LzopCodec.java:202: cannot find symbol
  •     [javac] symbol  : class BlockDecompressorStream
  •     [javac] location: class com.hadoop.compression.lzo.LzopCodec
  •     [javac]   protected static class LzopInputStream extends BlockDecompressorStream {
  •     [javac]                                                  ^
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/compression/lzo/LzopCodec.java:213: cannot find symbol
  •     [javac] symbol  : class Decompressor
  •     [javac] location: class com.hadoop.compression.lzo.LzopCodec.LzopInputStream
  •     [javac]     public LzopInputStream(InputStream in, Decompressor decompressor,
  •     [javac]                                            ^
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/mapreduce/LzoLineRecordReader.java:22: package org.apache.hadoop.conf does not exist
  •     [javac] import org.apache.hadoop.conf.Configuration;
  •     [javac]                              ^
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/mapreduce/LzoLineRecordReader.java:23: package org.apache.hadoop.fs does not exist
  •     [javac] import org.apache.hadoop.fs.FSDataInputStream;
  •    
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/mapreduce/LzoTextInputFormat.java:156: cannot find symbol
  •     [javac] symbol  : class FileSystem
  •     [javac] location: class com.hadoop.mapreduce.LzoTextInputFormat
  •     [javac]   private LzoIndex readIndex(Path file, FileSystem fs) throws IOException {
  •     [javac]                                         ^
  •     [javac] /soft/hadoop-gpl-compression-0.1.0/src/java/com/hadoop/mapreduce/LzoTextInputFormat.java:190: cannot find symbol
  •     [javac] symbol  : class FileSystem
  •     [javac] location: class com.hadoop.mapreduce.LzoTextInputFormat
  •     [javac]   public static void createIndex(FileSystem fs, Path lzoFile)
  •     [javac]                                  ^
  •     [javac] 100 errors
  • BUILD FAILED
  • /soft/hadoop-gpl-compression-0.1.0/build.xml:191: Compile failed; see the compiler error output for details.
  • Total time: 10 seconds
  • [root@linux-1 hadoop-gpl-compression-0.1.0]# [/ol]复制代码
    请问各位，可能是什么原因吗？吗？吗？吗？
    哪位有成功的经验，麻烦传道一下，呵呵。3q！！！
    • 回复

    使用道具 举报

    自定义广告语

    已有(7)人评论

    跳转到指定楼层
    正序浏览
    datong838 发表于 2013-10-25 10:46:11
    参考资料：
    http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ
    http://www.cloudera.com/blog/200 ... le-lzo-compression/
    http://labs.chinamobile.com/groups/10216_24274
    http://labs.chinamobile.com/groups/10216_24273
    回复

    使用道具 举报

    dgxl 发表于 2013-10-25 10:46:11
    是否需要指定hadoop的安装目录了吗？
    回复

    使用道具 举报

    llike90 发表于 2013-10-25 10:46:11
    我还价变量里面有HADOOP_HOME呢
    [root@linux-1 ~]# env |grep -i HOME
    HADOOP_HOME=/home/hadoop/hadoop
    PATH=.:/home/hadoop/jdk1.6.0_18_32/bin:/home/hadoop/hive/bin:/home/hadoop/hadoop/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/rrdtool/bin:/home/hadoop/apache-ant-1.8.0RC1/bin:/root/bin
    HIVE_HOME=/home/hadoop/hive
    JAVA_HOME=/home/hadoop/jdk1.6.0_18_32
    HOME=/root
    CLASSPATH=/home/hadoop/jdk1.6.0_18_32/lib/tools.jar:/home/hadoop/jdk1.6.0_18_32/lib/dt.jar:/home/hadoop/jdk1.6.0_18_32/lib/rt.jar:/home/hadoop/jdk1.6.0_18_32/lib:/home/hadoop/hadoop/lib::/usr/lib64:/usr/lib
    [root@linux-1 ~]#
    回复

    使用道具 举报

    leo_1989 发表于 2013-10-25 10:46:11
    Linux系统，不需要重新编译hadoop-gpl-compression。
    hadoop-gpl-compression包中有编译好的jar、so文件，拿来用就是了。
    回复

    使用道具 举报

    goldtimes 发表于 2013-10-25 10:46:11
    那么楼上兄台，hadoop的那些配置文件需要配置呢，配置内容为啥吗？
    麻烦给个例子，谢谢。
    回复

    使用道具 举报

    oYaoXiang1 发表于 2013-10-25 10:46:11
    看你好像用的32位linux，以32位linux举例：
    把hadoop-gpl-compression-0.1.0/hadoop-gpl-compression-0.1.0.jar copy到$HADOOP_HOME/lib目录下。把hadoop-gpl-compression-0.1.0/lib/native/Linux-i386-32的所有文件copy到$HADOOP_HOME/lib/native/Linux-i386-32目录下。
    编辑$HADOOP_CONF_DIR/hadoop-env.sh文件，修改HADOOP_OPTS行为
    export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native/Linux-i386-32"
    编辑$HADOOP_CONF_DIR/core-site.xml文件，追加
      
        io.compression.codecs
        org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec
        true
      
      
        io.compression.codec.lzo.class
        com.hadoop.compression.lzo.LzoCodec
        true
      
    重新启动hadoop。
    回复

    使用道具 举报

    jooli 发表于 2013-10-25 10:46:11
    忘说了，还需要安装 lzo2 rpm
    回复

    使用道具 举报

    高级模式
    B Color Image Link Quote Code Smilies
    您需要登录后才可以回帖 登录 | 立即注册

    本版积分规则

    发表新帖
    a2325145

    中级会员

    关注

    • 46

      主题

    • 137

      帖子

    • 3

      粉丝

    TA的主题
    关闭

    推荐上一条 /2 下一条

    关于我们· 联系我们· 加入我们· 小黑屋· 合作伙伴   

    Copyright © 2001-2024 About云-梭伦科技 Powered by Discuz! X3.4 Licensed Discuz Team.

    简书 / Smrz 京ICP备2020039040号 Wxb 简书网举报电话：021-34700000