专题 › Hive › 通过自动化shell脚本,每日定时执行导入hive数据仓库

Alkaloid0515 发表于 2015-5-11 20:39:33

通过自动化shell脚本,每日定时执行导入hive数据仓库

本帖最后由 pig2 于 2018-4-29 10:31 编辑

每日定时导入hive数据仓库的自动化脚本


创建shell脚本，创建临时表，装载数据，转换到正式的分区表中：
#!/bin/sh
# upload logs to hdfs

yesterday=`date --date='1 days ago' +%Y%m%d`

hive -e "
use stage;
create table tracklog_tmp (
dateday string,
datetime string,
ip string ,
cookieid string,
userid string,
logserverip string,
referer string,
requesturl string,
remark1 string,
remark2 string,
alexaflag string,
ua string,
wirelessflag string
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ' ';"


hive -e "
use stage;
set hive.enforce.bucketing=true;
set hive.exec.compress.output=true;
set mapred.output.compress=true;
set mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;
set io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec;
load data local inpath '/diskg/logs/tracklog_192.168.1.1/${yesterday}/${yesterday}????.dat' overwrite into table tracklog_tmp;
insert into table tracklog PARTITION (day='${yesterday}')select*from tracklog_tmp;
load data local inpath '/diskg/logs/tracklog_192.168.1.2/${yesterday}/${yesterday}????.dat' overwrite into table tracklog_tmp;
insert into table tracklog PARTITION (day='${yesterday}')select*from tracklog_tmp;
load data local inpath '/diskg/logs/tracklog_192.168.1.3/${yesterday}/${yesterday}????.dat' overwrite into table tracklog_tmp;
insert into table tracklog PARTITION (day='${yesterday}')select*from tracklog_tmp;
load data local inpath '/diskg/logs/trackloguc_192.168.1.1/${yesterday}/${yesterday}????.dat' overwrite into table tracklog_tmp;
insert into table tracklog PARTITION (day='${yesterday}')select*from tracklog_tmp;
load data local inpath '/diskg/logs/trackloguc_192.168.1.2/${yesterday}/${yesterday}????.dat' overwrite into table tracklog_tmp;
insert into table tracklog PARTITION (day='${yesterday}')select*from tracklog_tmp;
load data local inpath '/diskg/logs/trackloguc_192.168.1.3/${yesterday}/${yesterday}????.dat' overwrite into table tracklog_tmp;
insert into table tracklog PARTITION (day='${yesterday}')select*from tracklog_tmp;
"

hive -e "
use stage;
drop table tracklog_tmp ;"

在crontab中加入定时任务crontab -e 加入如下代码#import tracklog
2507 * * * /opt/bin/hive_opt/import_tracklog.sh

arBen 发表于 2015-5-12 08:34:03

不错不错....还是很简单的.

levycui 发表于 2015-5-12 09:17:35

谢谢楼主分享，这些实战的内容必须支持

吖louis 发表于 2015-5-12 10:57:06

{:soso_e182:}

hahaxixi 发表于 2015-5-12 13:36:32

感谢分享~~~~

jlon 发表于 2015-9-18 17:54:12

${yesterday}????.dat.这些问号是什么意思？？

地球仪 发表于 2016-8-30 16:41:58

很快就能用到了，不知道好用不！

查看完整版本: 通过自动化shell脚本,每日定时执行导入hive数据仓库