图文精华

About云»专题 › 技术学习(版主发帖区) › 大数据学习 › 实操演练› Mahout分步式程序开发 聚类Kmeans
分享

Mahout分步式程序开发 聚类Kmeans

52Pig 2014-10-28 21:20:40 发表于 实操演练 [显示全部楼层] 只看大图 回帖奖励 阅读模式 关闭右栏 3 9709
本帖最后由 52Pig 于 2014-10-28 22:18 编辑
阅读导读：
1.什么是聚类分析？
2.Mahout中的kmeans算法，默认的分融符是什么？
3.用kmeans算法得到的结果有什么特点？




1. 聚类算法kmeans
  聚类分析是数据挖掘及机器学习领域内的重点问题之一，在数据挖掘、模式识别、决策支持、机器学习及图像分割等领域有广泛的应用，是最重要的数据分析方法之一。聚类是在给定的数据集合中寻找同类的数据子集合，每一个子集合形成一个类簇，同类簇中的数据具有更大的相似性。聚类算法大体上可分为基于划分的方法、基于层次的方法、基于密度的方法、基于网格的方法以及基于模型的方法。
  k-means algorithm算法是一种得到最广泛使用的基于划分的聚类算法，把n个对象分为k个簇，以使簇内具有较高的相似度。相似度的计算根据一个簇中对象的平均值来进行。它与处理混合正态分布的最大期望算法很相似，因为他们都试图找到数据中自然聚类的中心。
  算法首先随机地选择k个对象，每个对象初始地代表了一个簇的平均值或中心。对剩余的每个对象根据其与各个簇中心的距离，将它赋给最近的簇，然后重新计算每个簇的平均值。这个过程不断重复，直到准则函数收敛。

2. Mahout开发环境介绍
  接上一篇文章：Mahout分步式程序开发 基于物品的协同过滤ItemCF。所有环境变量和系统配置与上文一致！

3. 用Mahout实现聚类算法kmeans
实现步骤:
  • 准备数据文件: randomData.csv
  • Java程序：KmeansHadoop.java
  • 运行程序
  • 聚类结果解读
  • HDFS产生的目录

1). 准备数据文件: randomData.csv
  数据文件randomData.csv，由R语言通过“随机正太分布函数”程序生成，单机内存实验请参考文章：用Maven构建Mahout项目。原始数据文件：这里只截取了一部分数据。
  1. 

  2. ~ vi datafile/randomData.csv

  3. -0.883033363823402 -3.31967192630249

  4. -2.39312626419456 3.34726861118871

  5. 2.66976353341256 1.85144276077058

  6. -1.09922906899594 -6.06261735207489

  7. -4.36361936997216 1.90509905380532

  8. -0.00351835125495037 -0.610105996559153

  9. -2.9962958796338 -3.60959839525735

  10. -3.27529418132066 0.0230099799641799

  11. 2.17665594420569 6.77290756817957

  12. -2.47862038335637 2.53431833167278

  13. 5.53654901906814 2.65089785582474

  14. 5.66257474538338 6.86783609641077

  15. -0.558946883114376 1.22332819416237

  16. 5.11728525486132 3.74663871584768

  17. 1.91240516693351 2.95874731384062

  18. -2.49747101306535 2.05006504756875

  19. 3.98781883213459 1.00780938946366

  20. 5.47470532716682 5.35084411045171
复制代码
  注：由于Mahout中kmeans算法，默认的分融符是” “(空格)，因些我把逗号分隔的数据文件，改成以空格分隔。2). Java程序：KmeansHadoop.javakmeans的算法实现，请查看Mahout in Action。
mahout-kmeans-process.png 


  1. 

  2. package org.conan.mymahout.cluster08;

  3. 

  4. import org.apache.hadoop.fs.Path;

  5. import org.apache.hadoop.mapred.JobConf;

  6. import org.apache.mahout.clustering.conversion.InputDriver;

  7. import org.apache.mahout.clustering.kmeans.KMeansDriver;

  8. import org.apache.mahout.clustering.kmeans.RandomSeedGenerator;

  9. import org.apache.mahout.common.distance.DistanceMeasure;

  10. import org.apache.mahout.common.distance.EuclideanDistanceMeasure;

  11. import org.apache.mahout.utils.clustering.ClusterDumper;

  12. import org.conan.mymahout.hdfs.HdfsDAO;

  13. import org.conan.mymahout.recommendation.ItemCFHadoop;

  14. 

  15. public class KmeansHadoop {

  16. private static final String HDFS = "hdfs://192.168.1.210:9000";

  17. 

  18. public static void main(String[] args) throws Exception {

  19. String localFile = "datafile/randomData.csv";

  20. String inPath = HDFS + "/user/hdfs/mix_data";

  21. String seqFile = inPath + "/seqfile";

  22. String seeds = inPath + "/seeds";

  23. String outPath = inPath + "/result/";

  24. String clusteredPoints = outPath + "/clusteredPoints";

  25. 

  26. JobConf conf = config();

  27. HdfsDAO hdfs = new HdfsDAO(HDFS, conf);

  28. hdfs.rmr(inPath);

  29. hdfs.mkdirs(inPath);

  30. hdfs.copyFile(localFile, inPath);

  31. hdfs.ls(inPath);

  32. 

  33. InputDriver.runJob(new Path(inPath), new Path(seqFile), "org.apache.mahout.math.RandomAccessSparseVector");

  34. 

  35. int k = 3;

  36. Path seqFilePath = new Path(seqFile);

  37. Path clustersSeeds = new Path(seeds);

  38. DistanceMeasure measure = new EuclideanDistanceMeasure();

  39. clustersSeeds = RandomSeedGenerator.buildRandom(conf, seqFilePath, clustersSeeds, k, measure);

  40. KMeansDriver.run(conf, seqFilePath, clustersSeeds, new Path(outPath), measure, 0.01, 10, true, 0.01, false);

  41. 

  42. Path outGlobPath = new Path(outPath, "clusters-*-final");

  43. Path clusteredPointsPath = new Path(clusteredPoints);

  44. System.out.printf("Dumping out clusters from clusters: %s and clusteredPoints: %s\n", outGlobPath, clusteredPointsPath);

  45. 

  46. ClusterDumper clusterDumper = new ClusterDumper(outGlobPath, clusteredPointsPath);

  47. clusterDumper.printClusters(null);

  48. }

  49. 

  50. public static JobConf config() {

  51. JobConf conf = new JobConf(ItemCFHadoop.class);

  52. conf.setJobName("ItemCFHadoop");

  53. conf.addResource("classpath:/hadoop/core-site.xml");

  54. conf.addResource("classpath:/hadoop/hdfs-site.xml");

  55. conf.addResource("classpath:/hadoop/mapred-site.xml");

  56. return conf;

  57. }

  58. }
复制代码
3). 运行程序
控制台输出:
  1. 

  2. Delete: hdfs://192.168.1.210:9000/user/hdfs/mix_data

  3. Create: hdfs://192.168.1.210:9000/user/hdfs/mix_data

  4. copy from: datafile/randomData.csv to hdfs://192.168.1.210:9000/user/hdfs/mix_data

  5. ls: hdfs://192.168.1.210:9000/user/hdfs/mix_data

  6. ==========================================================

  7. name: hdfs://192.168.1.210:9000/user/hdfs/mix_data/randomData.csv, folder: false, size: 36655

  8. ==========================================================

  9. SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

  10. SLF4J: Defaulting to no-operation (NOP) logger implementation

  11. SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

  12. 2013-10-14 15:39:31 org.apache.hadoop.util.NativeCodeLoader 

  13. 警告: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

  14. 2013-10-14 15:39:31 org.apache.hadoop.mapred.JobClient copyAndConfigureFiles

  15. 警告: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

  16. 2013-10-14 15:39:31 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus

  17. 信息: Total input paths to process : 1

  18. 2013-10-14 15:39:31 org.apache.hadoop.io.compress.snappy.LoadSnappy 

  19. 警告: Snappy native library not loaded

  20. 2013-10-14 15:39:31 org.apache.hadoop.mapred.JobClient monitorAndPrintJob

  21. 信息: Running job: job_local_0001

  22. 2013-10-14 15:39:31 org.apache.hadoop.mapred.Task initialize

  23. 信息: Using ResourceCalculatorPlugin : null

  24. 2013-10-14 15:39:31 org.apache.hadoop.mapred.Task done

  25. 信息: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting

  26. 2013-10-14 15:39:31 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate

  27. 信息: 

  28. 2013-10-14 15:39:31 org.apache.hadoop.mapred.Task commit

  29. 信息: Task attempt_local_0001_m_000000_0 is allowed to commit now

  30. 2013-10-14 15:39:31 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask

  31. 信息: Saved output of task 'attempt_local_0001_m_000000_0' to hdfs://192.168.1.210:9000/user/hdfs/mix_data/seqfile

  32. 2013-10-14 15:39:31 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate

  33. 信息: 

  34. 2013-10-14 15:39:31 org.apache.hadoop.mapred.Task sendDone

  35. 信息: Task 'attempt_local_0001_m_000000_0' done.

  36. 2013-10-14 15:39:32 org.apache.hadoop.mapred.JobClient monitorAndPrintJob

  37. 信息: map 100% reduce 0%

  38. 2013-10-14 15:39:32 org.apache.hadoop.mapred.JobClient monitorAndPrintJob

  39. 信息: Job complete: job_local_0001

  40. ......

  41. 2013-10-14 15:39:41 org.apache.hadoop.mapred.Merger$MergeQueue merge

  42. 信息: Down to the last merge-pass, with 1 segments left of total size: 677 bytes

  43. 2013-10-14 15:39:41 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate

  44. 信息: 

  45. 2013-10-14 15:39:41 org.apache.hadoop.mapred.Task done

  46. 信息: Task:attempt_local_0009_r_000000_0 is done. And is in the process of commiting

  47. 2013-10-14 15:39:41 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate

  48. 信息: 

  49. 2013-10-14 15:39:41 org.apache.hadoop.mapred.Task commit

  50. 信息: Task attempt_local_0009_r_000000_0 is allowed to commit now

  51. 2013-10-14 15:39:41 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask

  52. 信息: Saved output of task 'attempt_local_0009_r_000000_0' to hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusters-8

  53. 2013-10-14 15:39:41 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate

  54. 信息: reduce > reduce

  55. 2013-10-14 15:39:41 org.apache.hadoop.mapred.Task sendDone

  56. 信息: Task 'attempt_local_0009_r_000000_0' done.

  57. 2013-10-14 15:39:42 org.apache.hadoop.mapred.JobClient monitorAndPrintJob

  58. 信息: map 100% reduce 100%

  59. 2013-10-14 15:39:42 org.apache.hadoop.mapred.JobClient monitorAndPrintJob

  60. 信息: Job complete: job_local_0009

  61. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  62. 信息: Counters: 19

  63. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  64. 信息: File Output Format Counters 

  65. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  66. 信息: Bytes Written=695

  67. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  68. 信息: FileSystemCounters

  69. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  70. 信息: FILE_BYTES_READ=27256775

  71. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  72. 信息: HDFS_BYTES_READ=673669

  73. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  74. 信息: FILE_BYTES_WRITTEN=28569192

  75. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  76. 信息: HDFS_BYTES_WRITTEN=152767

  77. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  78. 信息: File Input Format Counters 

  79. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  80. 信息: Bytes Read=31390

  81. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  82. 信息: Map-Reduce Framework

  83. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  84. 信息: Map output materialized bytes=681

  85. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  86. 信息: Map input records=1000

  87. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  88. 信息: Reduce shuffle bytes=0

  89. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  90. 信息: Spilled Records=6

  91. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  92. 信息: Map output bytes=666

  93. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  94. 信息: Total committed heap usage (bytes)=1772093440

  95. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  96. 信息: SPLIT_RAW_BYTES=130

  97. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  98. 信息: Combine input records=0

  99. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  100. 信息: Reduce input records=3

  101. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  102. 信息: Reduce input groups=3

  103. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  104. 信息: Combine output records=0

  105. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  106. 信息: Reduce output records=3

  107. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Counters log

  108. 信息: Map output records=3

  109. 2013-10-14 15:39:42 org.apache.hadoop.mapred.JobClient copyAndConfigureFiles

  110. 警告: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

  111. 2013-10-14 15:39:42 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus

  112. 信息: Total input paths to process : 1

  113. 2013-10-14 15:39:42 org.apache.hadoop.mapred.JobClient monitorAndPrintJob

  114. 信息: Running job: job_local_0010

  115. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Task initialize

  116. 信息: Using ResourceCalculatorPlugin : null

  117. 2013-10-14 15:39:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer 

  118. 信息: io.sort.mb = 100

  119. 2013-10-14 15:39:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer 

  120. 信息: data buffer = 79691776/99614720

  121. 2013-10-14 15:39:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer 

  122. 信息: record buffer = 262144/327680

  123. 2013-10-14 15:39:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush

  124. 信息: Starting flush of map output

  125. 2013-10-14 15:39:42 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill

  126. 信息: Finished spill 0

  127. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Task done

  128. 信息: Task:attempt_local_0010_m_000000_0 is done. And is in the process of commiting

  129. 2013-10-14 15:39:42 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate

  130. 信息: 

  131. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Task sendDone

  132. 信息: Task 'attempt_local_0010_m_000000_0' done.

  133. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Task initialize

  134. 信息: Using ResourceCalculatorPlugin : null

  135. 2013-10-14 15:39:42 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate

  136. 信息: 

  137. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Merger$MergeQueue merge

  138. 信息: Merging 1 sorted segments

  139. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Merger$MergeQueue merge

  140. 信息: Down to the last merge-pass, with 1 segments left of total size: 677 bytes

  141. 2013-10-14 15:39:42 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate

  142. 信息: 

  143. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Task done

  144. 信息: Task:attempt_local_0010_r_000000_0 is done. And is in the process of commiting

  145. 2013-10-14 15:39:42 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate

  146. 信息: 

  147. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Task commit

  148. 信息: Task attempt_local_0010_r_000000_0 is allowed to commit now

  149. 2013-10-14 15:39:42 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask

  150. 信息: Saved output of task 'attempt_local_0010_r_000000_0' to hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusters-9

  151. 2013-10-14 15:39:42 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate

  152. 信息: reduce > reduce

  153. 2013-10-14 15:39:42 org.apache.hadoop.mapred.Task sendDone

  154. 信息: Task 'attempt_local_0010_r_000000_0' done.

  155. 2013-10-14 15:39:43 org.apache.hadoop.mapred.JobClient monitorAndPrintJob

  156. 信息: map 100% reduce 100%

  157. 2013-10-14 15:39:43 org.apache.hadoop.mapred.JobClient monitorAndPrintJob

  158. 信息: Job complete: job_local_0010

  159. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  160. 信息: Counters: 19

  161. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  162. 信息: File Output Format Counters 

  163. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  164. 信息: Bytes Written=695

  165. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  166. 信息: FileSystemCounters

  167. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  168. 信息: FILE_BYTES_READ=30544993

  169. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  170. 信息: HDFS_BYTES_READ=741007

  171. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  172. 信息: FILE_BYTES_WRITTEN=32013760

  173. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  174. 信息: HDFS_BYTES_WRITTEN=154545

  175. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  176. 信息: File Input Format Counters 

  177. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  178. 信息: Bytes Read=31390

  179. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  180. 信息: Map-Reduce Framework

  181. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  182. 信息: Map output materialized bytes=681

  183. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  184. 信息: Map input records=1000

  185. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  186. 信息: Reduce shuffle bytes=0

  187. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  188. 信息: Spilled Records=6

  189. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  190. 信息: Map output bytes=666

  191. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  192. 信息: Total committed heap usage (bytes)=1966735360

  193. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  194. 信息: SPLIT_RAW_BYTES=130

  195. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  196. 信息: Combine input records=0

  197. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  198. 信息: Reduce input records=3

  199. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  200. 信息: Reduce input groups=3

  201. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  202. 信息: Combine output records=0

  203. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  204. 信息: Reduce output records=3

  205. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Counters log

  206. 信息: Map output records=3

  207. 2013-10-14 15:39:43 org.apache.hadoop.mapred.JobClient copyAndConfigureFiles

  208. 警告: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

  209. 2013-10-14 15:39:43 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus

  210. 信息: Total input paths to process : 1

  211. 2013-10-14 15:39:43 org.apache.hadoop.mapred.JobClient monitorAndPrintJob

  212. 信息: Running job: job_local_0011

  213. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Task initialize

  214. 信息: Using ResourceCalculatorPlugin : null

  215. 2013-10-14 15:39:43 org.apache.hadoop.mapred.MapTask$MapOutputBuffer 

  216. 信息: io.sort.mb = 100

  217. 2013-10-14 15:39:43 org.apache.hadoop.mapred.MapTask$MapOutputBuffer 

  218. 信息: data buffer = 79691776/99614720

  219. 2013-10-14 15:39:43 org.apache.hadoop.mapred.MapTask$MapOutputBuffer 

  220. 信息: record buffer = 262144/327680

  221. 2013-10-14 15:39:43 org.apache.hadoop.mapred.MapTask$MapOutputBuffer flush

  222. 信息: Starting flush of map output

  223. 2013-10-14 15:39:43 org.apache.hadoop.mapred.MapTask$MapOutputBuffer sortAndSpill

  224. 信息: Finished spill 0

  225. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Task done

  226. 信息: Task:attempt_local_0011_m_000000_0 is done. And is in the process of commiting

  227. 2013-10-14 15:39:43 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate

  228. 信息: 

  229. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Task sendDone

  230. 信息: Task 'attempt_local_0011_m_000000_0' done.

  231. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Task initialize

  232. 信息: Using ResourceCalculatorPlugin : null

  233. 2013-10-14 15:39:43 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate

  234. 信息: 

  235. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Merger$MergeQueue merge

  236. 信息: Merging 1 sorted segments

  237. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Merger$MergeQueue merge

  238. 信息: Down to the last merge-pass, with 1 segments left of total size: 677 bytes

  239. 2013-10-14 15:39:43 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate

  240. 信息: 

  241. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Task done

  242. 信息: Task:attempt_local_0011_r_000000_0 is done. And is in the process of commiting

  243. 2013-10-14 15:39:43 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate

  244. 信息: 

  245. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Task commit

  246. 信息: Task attempt_local_0011_r_000000_0 is allowed to commit now

  247. 2013-10-14 15:39:43 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask

  248. 信息: Saved output of task 'attempt_local_0011_r_000000_0' to hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusters-10

  249. 2013-10-14 15:39:43 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate

  250. 信息: reduce > reduce

  251. 2013-10-14 15:39:43 org.apache.hadoop.mapred.Task sendDone

  252. 信息: Task 'attempt_local_0011_r_000000_0' done.

  253. 2013-10-14 15:39:44 org.apache.hadoop.mapred.JobClient monitorAndPrintJob

  254. 信息: map 100% reduce 100%

  255. 2013-10-14 15:39:44 org.apache.hadoop.mapred.JobClient monitorAndPrintJob

  256. 信息: Job complete: job_local_0011

  257. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  258. 信息: Counters: 19

  259. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  260. 信息: File Output Format Counters 

  261. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  262. 信息: Bytes Written=695

  263. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  264. 信息: FileSystemCounters

  265. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  266. 信息: FILE_BYTES_READ=33833211

  267. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  268. 信息: HDFS_BYTES_READ=808345

  269. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  270. 信息: FILE_BYTES_WRITTEN=35458320

  271. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  272. 信息: HDFS_BYTES_WRITTEN=156323

  273. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  274. 信息: File Input Format Counters 

  275. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  276. 信息: Bytes Read=31390

  277. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  278. 信息: Map-Reduce Framework

  279. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  280. 信息: Map output materialized bytes=681

  281. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  282. 信息: Map input records=1000

  283. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  284. 信息: Reduce shuffle bytes=0

  285. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  286. 信息: Spilled Records=6

  287. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  288. 信息: Map output bytes=666

  289. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  290. 信息: Total committed heap usage (bytes)=2166095872

  291. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  292. 信息: SPLIT_RAW_BYTES=130

  293. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  294. 信息: Combine input records=0

  295. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  296. 信息: Reduce input records=3

  297. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  298. 信息: Reduce input groups=3

  299. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  300. 信息: Combine output records=0

  301. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  302. 信息: Reduce output records=3

  303. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Counters log

  304. 信息: Map output records=3

  305. 2013-10-14 15:39:44 org.apache.hadoop.mapred.JobClient copyAndConfigureFiles

  306. 警告: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

  307. 2013-10-14 15:39:44 org.apache.hadoop.mapreduce.lib.input.FileInputFormat listStatus

  308. 信息: Total input paths to process : 1

  309. 2013-10-14 15:39:44 org.apache.hadoop.mapred.JobClient monitorAndPrintJob

  310. 信息: Running job: job_local_0012

  311. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Task initialize

  312. 信息: Using ResourceCalculatorPlugin : null

  313. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Task done

  314. 信息: Task:attempt_local_0012_m_000000_0 is done. And is in the process of commiting

  315. 2013-10-14 15:39:44 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate

  316. 信息: 

  317. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Task commit

  318. 信息: Task attempt_local_0012_m_000000_0 is allowed to commit now

  319. 2013-10-14 15:39:44 org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter commitTask

  320. 信息: Saved output of task 'attempt_local_0012_m_000000_0' to hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusteredPoints

  321. 2013-10-14 15:39:44 org.apache.hadoop.mapred.LocalJobRunner$Job statusUpdate

  322. 信息: 

  323. 2013-10-14 15:39:44 org.apache.hadoop.mapred.Task sendDone

  324. 信息: Task 'attempt_local_0012_m_000000_0' done.

  325. 2013-10-14 15:39:45 org.apache.hadoop.mapred.JobClient monitorAndPrintJob

  326. 信息: map 100% reduce 0%

  327. 2013-10-14 15:39:45 org.apache.hadoop.mapred.JobClient monitorAndPrintJob

  328. 信息: Job complete: job_local_0012

  329. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log

  330. 信息: Counters: 11

  331. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log

  332. 信息: File Output Format Counters 

  333. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log

  334. 信息: Bytes Written=41520

  335. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log

  336. 信息: File Input Format Counters 

  337. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log

  338. 信息: Bytes Read=31390

  339. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log

  340. 信息: FileSystemCounters

  341. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log

  342. 信息: FILE_BYTES_READ=18560374

  343. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log

  344. 信息: HDFS_BYTES_READ=437203

  345. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log

  346. 信息: FILE_BYTES_WRITTEN=19450325

  347. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log

  348. 信息: HDFS_BYTES_WRITTEN=120417

  349. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log

  350. 信息: Map-Reduce Framework

  351. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log

  352. 信息: Map input records=1000

  353. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log

  354. 信息: Spilled Records=0

  355. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log

  356. 信息: Total committed heap usage (bytes)=1083047936

  357. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log

  358. 信息: SPLIT_RAW_BYTES=130

  359. 2013-10-14 15:39:45 org.apache.hadoop.mapred.Counters log

  360. 信息: Map output records=1000

  361. Dumping out clusters from clusters: hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusters-*-final and clusteredPoints: hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusteredPoints

  362. CL-552{n=443 c=[1.631, -0.412] r=[1.563, 1.407]}

  363. Weight : [props - optional]: Point:

  364. 1.0: [-2.393, 3.347]

  365. 1.0: [-4.364, 1.905]

  366. 1.0: [-3.275, 0.023]

  367. 1.0: [-2.479, 2.534]

  368. 1.0: [-0.559, 1.223]

  369. ...

  370. 

  371. CL-847{n=77 c=[-2.953, -0.971] r=[1.767, 2.189]}

  372. Weight : [props - optional]: Point:

  373. 1.0: [-0.883, -3.320]

  374. 1.0: [-1.099, -6.063]

  375. 1.0: [-0.004, -0.610]

  376. 1.0: [-2.996, -3.610]

  377. 1.0: [3.988, 1.008]

  378. ...

  379. 

  380. CL-823{n=480 c=[0.219, 2.600] r=[1.479, 1.385]}

  381. Weight : [props - optional]: Point:

  382. 1.0: [2.670, 1.851]

  383. 1.0: [2.177, 6.773]

  384. 1.0: [5.537, 2.651]

  385. 1.0: [5.663, 6.868]

  386. 1.0: [5.117, 3.747]

  387. 1.0: [1.912, 2.959]

  388. ...
复制代码
4). 聚类结果解读
我们可以把上面的日志分解析成3个部分解读
  • 初始化环境
  • 算法执行
  • 打印聚类结果
a. 初始化环境
HDFS的数据目录和工作目录，并上传数据文件。
  1. Delete: hdfs://192.168.1.210:9000/user/hdfs/mix_data

  2. Create: hdfs://192.168.1.210:9000/user/hdfs/mix_data

  3. copy from: datafile/randomData.csv to hdfs://192.168.1.210:9000/user/hdfs/mix_data

  4. ls: hdfs://192.168.1.210:9000/user/hdfs/mix_data

  5. ==========================================================

  6. name: hdfs://192.168.1.210:9000/user/hdfs/mix_data/randomData.csv, folder: false, size: 36655
复制代码
b. 算法执行
算法执行，有3个步骤。
  • 把原始数据randomData.csv，转成Mahout sequence files of VectorWritable。
  • 通过随机的方法，选中kmeans的3个中心，做为初始集群 。
  • 根据迭代次数的设置，执行MapReduce，进行计算。
1):把原始数据randomData.csv，转成Mahout sequence files of VectorWritable。  程序源代码:
  1. InputDriver.runJob(new Path(inPath), new Path(seqFile), "org.apache.mahout.math.RandomAccessSparseVector");
复制代码

日志输出：
Job complete: job_local_0001
2):通过随机的方法，选中kmeans的3个中心，做为初始集群 程序源代码:
  1. int k = 3;

  2. Path seqFilePath = new Path(seqFile);

  3. Path clustersSeeds = new Path(seeds);

  4. DistanceMeasure measure = new EuclideanDistanceMeasure();

  5. clustersSeeds = RandomSeedGenerator.buildRandom(conf, seqFilePath, clustersSeeds, k, measure);
复制代码
日志输出：
  1. Job complete: job_local_0002
复制代码
3):根据迭代次数的设置，执行MapReduce，进行计算
程序源代码:
  1. KMeansDriver.run(conf, seqFilePath, clustersSeeds, new Path(outPath), measure, 0.01, 10, true, 0.01, false);
复制代码
日志输出：
  1. Job complete: job_local_0003

  2. Job complete: job_local_0004

  3. Job complete: job_local_0005

  4. Job complete: job_local_0006

  5. Job complete: job_local_0007

  6. Job complete: job_local_0008

  7. Job complete: job_local_0009

  8. Job complete: job_local_0010

  9. Job complete: job_local_0011

  10. Job complete: job_local_0012
复制代码
c.打印聚类结果
  1. Dumping out clusters from clusters: hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusters-*-final and clusteredPoints: hdfs://192.168.1.210:9000/user/hdfs/mix_data/result/clusteredPoints

  2. CL-552{n=443 c=[1.631, -0.412] r=[1.563, 1.407]}

  3. CL-847{n=77 c=[-2.953, -0.971] r=[1.767, 2.189]}

  4. CL-823{n=480 c=[0.219, 2.600] r=[1.479, 1.385]}
复制代码
运行结果：有3个中心。
  1. Cluster1, 包括443个点，中心坐标[1.631, -0.412]

  2. Cluster2, 包括77个点，中心坐标[-2.953, -0.971]

  3. Cluster3, 包括480 个点，中心坐标[0.219, 2.600]
复制代码
5). HDFS产生的目录
  1. # 根目录

  2. ~ hadoop fs -ls /user/hdfs/mix_data

  3. Found 4 items

  4. -rw-r--r-- 3 Administrator supergroup 36655 2013-10-04 15:31 /user/hdfs/mix_data/randomData.csv

  5. drwxr-xr-x - Administrator supergroup 0 2013-10-04 15:31 /user/hdfs/mix_data/result

  6. drwxr-xr-x - Administrator supergroup 0 2013-10-04 15:31 /user/hdfs/mix_data/seeds

  7. drwxr-xr-x - Administrator supergroup 0 2013-10-04 15:31 /user/hdfs/mix_data/seqfile

  8. 

  9. # 输出目录

  10. ~ hadoop fs -ls /user/hdfs/mix_data/result

  11. Found 13 items

  12. -rw-r--r-- 3 Administrator supergroup 194 2013-10-04 15:31 /user/hdfs/mix_data/result/_policy

  13. drwxr-xr-x - Administrator supergroup 0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusteredPoints

  14. drwxr-xr-x - Administrator supergroup 0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-0

  15. drwxr-xr-x - Administrator supergroup 0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-1

  16. drwxr-xr-x - Administrator supergroup 0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-10-final

  17. drwxr-xr-x - Administrator supergroup 0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-2

  18. drwxr-xr-x - Administrator supergroup 0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-3

  19. drwxr-xr-x - Administrator supergroup 0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-4

  20. drwxr-xr-x - Administrator supergroup 0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-5

  21. drwxr-xr-x - Administrator supergroup 0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-6

  22. drwxr-xr-x - Administrator supergroup 0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-7

  23. drwxr-xr-x - Administrator supergroup 0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-8

  24. drwxr-xr-x - Administrator supergroup 0 2013-10-04 15:31 /user/hdfs/mix_data/result/clusters-9

  25. 

  26. # 产生的随机中心种子目录

  27. ~ hadoop fs -ls /user/hdfs/mix_data/seeds

  28. Found 1 items

  29. -rw-r--r-- 3 Administrator supergroup 599 2013-10-04 15:31 /user/hdfs/mix_data/seeds/part-randomSeed

  30. 

  31. # 输入文件换成Mahout格式文件的目录

  32. ~ hadoop fs -ls /user/hdfs/mix_data/seqfile

  33. Found 2 items

  34. -rw-r--r-- 3 Administrator supergroup 0 2013-10-04 15:31 /user/hdfs/mix_data/seqfile/_SUCCESS

  35. -rw-r--r-- 3 Administrator supergroup 31390 2013-10-04 15:31 /user/hdfs/mix_data/seqfile/part-m-00000

复制代码
4. 用R语言可视化结果
  分别把聚类后的点，保存到不同的cluster*.csv文件，然后用R语言画图。
  1. c1<-read.csv(file="cluster1.csv",sep=",",header=FALSE)

  2. c2<-read.csv(file="cluster2.csv",sep=",",header=FALSE)

  3. c3<-read.csv(file="cluster3.csv",sep=",",header=FALSE)

  4. y<-rbind(c1,c2,c3)

  5. cols<-c(rep(1,nrow(c1)),rep(2,nrow(c2)),rep(3,nrow(c3)))

  6. plot(y, col=c("black","blue","green")[cols])

  7. center<-matrix(c(1.631, -0.412,-2.953, -0.971,0.219, 2.600),ncol=2,byrow=TRUE)

  8. points(center, col="violetred", pch = 19)
复制代码

   kmeans.png
从上图中，我们看到有 黑，蓝，绿，三种颜色的空心点，这些点就是原始数据。3个紫色实点，是Mahout的kmeans后生成的3个中心。
  对比文章中用R语言实现的kmeans的分类和中心，都不太一样。用Maven构建Mahout项目  简单总结一下，在使用kmeans时，根据距离算法，阈值，初始中心，迭代次数的不同，kmeans计算的结果是不相同的。
  因此，用kmeans算法，我们一般只能得到一个模糊的分类标准，这个标准对于我们认识未知领域的数据集是很有帮助的。不能做为精确衡量数据的指标。












回复

使用道具 举报

自定义广告语

已有(3)人评论

跳转到指定楼层
正序浏览
hahaxixi 发表于 2014-10-29 09:34:57
学习了，不错，很详细了~~~
回复

使用道具 举报

永无止进 发表于 2014-10-29 11:03:31
学习了，不错
回复

使用道具 举报

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

发表新帖
52Pig

中级会员

关注

  • 49

    主题

  • 57

    帖子

  • 16

    粉丝

TA的主题
关闭

推荐上一条 /2 下一条

关于我们· 联系我们· 加入我们· 小黑屋· 合作伙伴   

Copyright © 2001-2025 About云 Powered by Discuz! X3.4 Licensed Discuz Team.

简书 / Smrz 京ICP备2020039040号 Wxb 简书网举报电话：021-34700000