日志
MapReuce 怎么实现去掉文件重复的内容
热度 1 |
文件数据格式：
Name,CardNo,Descriot,CtfTp,CtfId,Gender,Birthday,Address
彬,,,ID,xxxxx21059,M,20101001,,,F,,CHN,,,,,,,,139,19:46:02,4000000
珊,,,ID,xxxxx811090189,M,20101001,,,F,,CHN,,,,,,,,158
世,,,ID,4xxxxx1772,M,19880129,,,F,,CHN,,,,,,,,152108,2,1,,,,,0,2010-10-19
超,,,ID,4xxxxx1772,M,20101001,,,F,,CHN,,,,,,,,1585091,,,mc06o.cn,,,,,,,,0,2010-10-19 19:46:04,4000003
磊,,,ID,4xxxxx1772,M,19831011,,,F,,CHN,,,,,,,,159159,362,,565408,,,,,,,,0,2010-10-19 19:46:05,4000004
芳,,,ID,xxxxx811090189,M,20101001,,,F,,CHN,,,,,,,,1506:05,4005
要求：读取文件后，我要根据CtfTp 字段删除对应的字段内容，删除掉重复的，只保留第一条。
删除后的数据格式应该是：
彬,,,ID,xxxxx21059,M,20101001,,,F,,CHN,,,,,,,,139,19:46:02,4000000
珊,,,ID,xxxxx811090189,M,20101001,,,F,,CHN,,,,,,,,158
世,,,ID,4xxxxx1772,M,19880129,,,F,,CHN,,,,,,,,152108,2,1,,,,,0,2010-10-19
Name,CardNo,Descriot,CtfTp,CtfId,Gender,Birthday,Address
彬,,,ID,xxxxx21059,M,20101001,,,F,,CHN,,,,,,,,139,19:46:02,4000000
珊,,,ID,xxxxx811090189,M,20101001,,,F,,CHN,,,,,,,,158
世,,,ID,4xxxxx1772,M,19880129,,,F,,CHN,,,,,,,,152108,2,1,,,,,0,2010-10-19
超,,,ID,4xxxxx1772,M,20101001,,,F,,CHN,,,,,,,,1585091,,,mc06o.cn,,,,,,,,0,2010-10-19 19:46:04,4000003
磊,,,ID,4xxxxx1772,M,19831011,,,F,,CHN,,,,,,,,159159,362,,565408,,,,,,,,0,2010-10-19 19:46:05,4000004
芳,,,ID,xxxxx811090189,M,20101001,,,F,,CHN,,,,,,,,1506:05,4005
要求：读取文件后，我要根据CtfTp 字段删除对应的字段内容，删除掉重复的，只保留第一条。
删除后的数据格式应该是：
彬,,,ID,xxxxx21059,M,20101001,,,F,,CHN,,,,,,,,139,19:46:02,4000000
珊,,,ID,xxxxx811090189,M,20101001,,,F,,CHN,,,,,,,,158
世,,,ID,4xxxxx1772,M,19880129,,,F,,CHN,,,,,,,,152108,2,1,,,,,0,2010-10-19
发表评论 评论 (2 个评论)
- 回复 nettman
- MapReduce初级案例（1）：使用MapReduce去重
http://www.aboutyun.com/thread-7041-1-1.html
- 回复 zhanggl
- 大哥，你这篇文章，这个地址一年前就有了。http://www.2cto.com/database/201305/209241.html
nettman: MapReduce初级案例（1）：使用MapReduce去重
http://www.aboutyun.com/thread-7041-1-1.html
/2 
