大数据环境启用LZO压缩
微信搜索关注：若泽大数据，有精美课程和定时推送技术博客，qq交流群：707635769，一起交流，一起进步
CentOS7中安装LZO压缩程序
一、准备工作：
yum -y installlzo-develzlib-develgcc autoconf automake libtool
二、安装LZO
1、解压编译，并安装
cd /opt/software
wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.09.tar.gz
tar -zxvf lzo-2.09.tar.gz
cd lzo-2.09
./configure -enable-shared -prefix=/usr/local/hadoop/lzo/
make && make test && make install
2、复制文件
将/usr/local/hadoop/lzo/lib/* 复制到/usr/lib/和/usr/lib64/下
cp /usr/local/hadoop/lzo/lib/* /usr/lib/
cp /usr/local/hadoop/lzo/lib/* /usr/lib64/
3、修改配置环境变量(vi ~/.bash_profile)，增加如下内容：
export PATH=/usr/local/hadoop/lzo/:$PATH
三、安装LZOP
1、下载并解压
cd /opt/software
wget http://www.lzop.org/download/lzop-1.04.tar.gz
tar -zxvf lzop-1.04.tar.gz
2、在编译前需要的环境变量(~/.bash_profile)中配置如下内容：
export C_INCLUDE_PATH=/usr/local/hadoop/lzo/include/
注：如不配置变量，在编译时会报：configure: error: LZO header files not found. Please check your installation or set the environment variable `CPPFLAGS'.
3、进入解压后目录，并编译安装
cd cd /opt/software/lzop-1.04
./configure -enable-shared -prefix=/usr/local/hadoop/lzop
make&& make install
4、将lzop复制到/usr/bin/
ln -s /usr/local/hadoop/lzop/bin/lzop /usr/bin/lzop
5、测试lzop
输入：lzop nohup.out
产生：lzo后缀的压缩文件： /home/hadoop/data/access_20131219.log.lzo即表示成功
注：在测试中可能遇到报错：lzop: error while loading shared libraries: liblzo2.so.2: cannot open shared object file: No such file or directory
解决办法：增加环境变量(~/.bash_profile)export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64
四、安装Hadoop-LZO
注：编译时需要maven，自行配置好maven
1、下载介质：https://github.com/twitter/hadoop-lzo
2、解压并编译：
cd /opt/software/hadoop-lzo-release-0.4.19
mvn clean package -Dmaven.test.skip=true
3、编译完成执行如下命令：
tar -cBf --C target/native/Linux-amd64-64/lib . | tar -xBvf --C /app/hadoop-2.6.0-cdh5.7.0/lib/native
cp target/hadoop-lzo-0.4.19.jar /app/hadoop-2.6.0-cdh5.7.0/share/hadoop/common/
如果为集群环境，则接下来就是将/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/common/hadoop-lzo-0.4.19.jar以及/app/hadoop-2.6.0-cdh5.7.0/lib/native/同步到其它所有的hadoop节点。
注意，要保证目录/app/hadoop-2.6.0-cdh5.7.0/lib/native/下的jar包，运行hadoop的用户都有执行权限。
五、产生index文件
cd /app/hadoop-2.6.0-cdh5.7.0/share/hadoop/common
hadoop jar hadoop-lzo-0.4.19.jar com.hadoop.compression.lzo.LzoIndexer /user/hive/warehouse/page_views_parquet1/page_views_parquet.lzo
注：lzo文件必须在hdfs文件系统中。
至此完成CentOS7中安装LZO压缩程序
配置Hadoop中启用LZO压缩
配置Hadoop中启用LZO压缩，并完成测试。步骤如下：
一、配置hadoop的hadoop-evn.sh文件，增加如下内容：
[*]export LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib
二、配置core-site.xml文件，增加如下内容：
[*]<!--支持的压缩列表-->
[*]<property>
[*] <name>io.compression.codecs</name>
[*] <value>
[*] org.apache.hadoop.io.compress.GzipCodec,
[*] org.apache.hadoop.io.compress.DefaultCodec,
[*] org.apache.hadoop.io.compress.BZip2Codec,
[*] org.apache.hadoop.io.compress.SnappyCodec,
[*] com.hadoop.compression.lzo.LzoCodec,
[*] com.hadoop.compression.lzo.LzopCodec
[*] </value>
[*]</property>
[*]
[*]
[*]<!--支持LZO使用类-->
[*]<property>
[*] <name>io.compression.codec.lzo.class</name>
[*] <value>com.hadoop.compression.lzo.LzopCodec</value>
[*]</property>
二、配置mapred-site.xml文件，增加如下内容：
[*]<!--启用map中间文件压缩-->
[*]<property>
[*] <name>mapreduce.map.output.compress</name>
[*] <value>true</value>
[*]</property>
[*]<!--启用map中间压缩类-->
[*]<property>
[*] <name>mapred.map.output.compression.codec</name>
[*] <value>com.hadoop.compression.lzo.LzopCodec</value>
[*]</property>
[*]<!--启用mapreduce文件压缩-->
[*]<property>
[*] <name>mapreduce.output.fileoutputformat.compress</name>
[*] <value>true</value>
[*]</property>
[*]<!--启用mapreduce压缩类-->
[*]<property>
[*] <name>mapreduce.output.fileoutputformat.compress.codec</name>
[*] <value>com.hadoop.compression.lzo.LzopCodec</value>
[*]</property>
[*]<!--配置Jar包-->
[*]<property>
[*] <name>mapred.child.env</name>
[*] <value>LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib</value>
[*]</property>
三、使用hadoop自带wordcount程序测试
1、测试生成lzo文件
[*]cd /app/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce
[*]hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar wordcount /input/test1.txt /output/wc2
测试结果：
$ hdfs dfs -ls/output/wc2
Found 2 items
-rw-r--r-- 1 hadoop supergroup 0 2018-03-17 00:21 /output/wc2/_SUCCESS
-rw-r--r-- 1 hadoop supergroup 113 2018-03-17 00:21 /output/wc2/part-r-00000.lzo
2、生成index文件：
[*]cd /app/hadoop-2.6.0-cdh5.7.0/share/hadoop/common
[*]hadoop jar hadoop-lzo-0.4.19.jar com.hadoop.compression.lzo.LzoIndexer /output/wc2/part-r-00000.lzo
日志：
18/03/17 00:23:05 INFO lzo.GPLNativeCodeLoader: Loaded native gpl libraryutput/wc2/part-r-00000.lzo
18/03/17 00:23:05 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library
18/03/17 00:23:06 INFO lzo.LzoIndexer: LZO Indexing file /output/wc2/part-r-00000.lzo, size 0.00 GB...
18/03/17 00:23:07 INFO Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
18/03/17 00:23:07 INFO lzo.LzoIndexer: Completed LZO Indexing in 0.80 seconds (0.00 MB/s).Index size is 0.01 KB.
测试结果：
$ hdfs dfs -ls/output/wc2
Found 3 items
-rw-r--r-- 1 hadoop supergroup 0 2018-03-17 00:21 /output/wc2/_SUCCESS
-rw-r--r-- 1 hadoop supergroup 113 2018-03-17 00:21 /output/wc2/part-r-00000.lzo
-rw-r--r-- 1 hadoop supergroup 8 2018-03-17 00:23 /output/wc2/part-r-00000.lzo.index
至此完成配置与测试
spark中配置启用LZO压缩
Spark中配置启用LZO压缩，步骤如下：
一、spark-env.sh配置
[*]export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/app/hadoop-2.6.0-cdh5.7.0/lib/native
[*]export SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/app/hadoop-2.6.0-cdh5.7.0/lib/native
[*]export SPARK_CLASSPATH=$SPARK_CLASSPATH:/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/yarn/*:/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/yarn/lib/*:/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/common/*:/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/common/lib/*:/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/hdfs/*:/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/hdfs/lib/*:/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce/*:/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce/lib/*:/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/tools/lib/*:/app/spark-2.2.0-bin-2.6.0-cdh5.7.0/jars/*
二、spark-defaults.conf配置
[*]spark.driver.extraClassPath /app/hadoop-2.6.0-cdh5.7.0/share/hadoop/common/hadoop-lzo-0.4.19.jar
[*]spark.executor.extraClassPath /app/hadoop-2.6.0-cdh5.7.0/share/hadoop/common/hadoop-lzo-0.4.19.jar
注：指向编译生成lzo的jar包
三、测试
1、读取Lzo文件
[*]spark-shell --master local
[*]scala> import com.hadoop.compression.lzo.LzopCodec
[*]scala> val page_views = sc.textFile("/user/hive/warehouse/page_views_lzo/page_views.dat.lzo")
2、写出lzo文件
[*]spark-shell --master local
[*]scala> import com.hadoop.compression.lzo.LzopCodec
[*]scala> val lzoTest = sc.parallelize(1 to 10)
[*]scala> lzoTest.saveAsTextFile("/input/test_lzo", classOf)
[*]结果：
[*]$ hdfs dfs -ls /input/test_lzo
Found 3 items
-rw-r--r-- 1 hadoop supergroup 0 2018-03-16 23:24 /input/test_lzo/_SUCCESS
-rw-r--r-- 1 hadoop supergroup 60 2018-03-16 23:24 /input/test_lzo/part-00000.lzo
-rw-r--r-- 1 hadoop supergroup 61 2018-03-16 23:24 /input/test_lzo/part-00001.lzo
至此配置与测试完成。
四、配置与测试中存问题
1、引用native，缺少LD_LIBRARY_PATH
1.1、错误提示：
[*]Caused by: java.lang.RuntimeException: native-lzo library not available
at com.hadoop.compression.lzo.LzopCodec.getDecompressorType(LzopCodec.java:120)
at org.apache.hadoop.io.compress.CodecPool.getDecompressor(CodecPool.java:178)
at org.apache.hadoop.mapred.LineRecordReader.(LineRecordReader.java:111)
at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
at org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1(HadoopRDD.scala:246)
at org.apache.spark.rdd.HadoopRDD$$anon$1.(HadoopRDD.scala:245)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:203)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:94)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[*]
1.2、解决办法：在spark的conf中配置spark-evn.sh，增加以下内容：
[*]export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/app/hadoop-2.6.0-cdh5.7.0/lib/native
export SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/app/hadoop-2.6.0-cdh5.7.0/lib/native
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/yarn/*:/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/yarn/lib/*:/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/common/*:/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/common/lib/*:/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/hdfs/*:/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/hdfs/lib/*:/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce/*:/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce/lib/*:/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/tools/lib/*:/app/spark-2.2.0-bin-2.6.0-cdh5.7.0/jars/*
2、无法找到LzopCodec类
2.1、错误提示：
[*]Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzopCodec not found.
[*] at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
[*] at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
[*] at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
[*]Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzopCodec not found
[*] at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980)
[*] at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
2.2、解决办法：在spark的conf中配置spark-defaults.conf，增加以下内容：
[*]spark.driver.extraClassPath /app/hadoop-2.6.0-cdh5.7.0/share/hadoop/common/hadoop-lzo-0.4.19.jar
[*]spark.executor.extraClassPath /app/hadoop-2.6.0-cdh5.7.0/share/hadoop/common/hadoop-lzo-0.4.19.jar
页:
[1]