hadoop2.4.1 QJM ha集群namenode的edits log文件清除问题，求大神帮忙看看

bob007 发表于 2015-8-7 13:39
合并应该是journal node，另外两个同步就可以了吧

合并在standy namenode上，journal node只负责存储

supertianxiang 发表于 2015-8-7 14:24
合并在standy namenode上，journal node只负责存储

如何实现fsimage和edits的合并？在standby NameNode节点上会一直运行一个叫做CheckpointerThread的线程，这个线程调用StandbyCheckpointer类的doWork()函数，而doWork函数会每隔Math.min(checkpointCheckPeriod, checkpointPeriod)秒来坐一次合并操作。

楼主的问题最终解决了吗？
缩小下时间，看看是否有变化。配置反正都是这样配的

jixianqiuxue 发表于 2015-8-7 17:18
如何实现fsimage和edits的合并？在standby NameNode节点上会一直运行一个叫做CheckpointerThread的线程 ...

问题基本解决，感谢！
从打印信息看
482000 2015-08-08 15:10:45,515 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Transfer took 0.86s at 19035.88 KB/s
482001 2015-08-08 15:10:45,516 INFO org.apache.hadoop.hdfs.server.namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000003751114 size 16842507 by
       tes.
482002 2015-08-08 15:10:45,668 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to retain 2 images with txid >= 3749021
482003 2015-08-08 15:10:45,669 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Purging old image FSImageFile(file=/export/distributed/
       hadoop/dfs/name/current/fsimage_0000000000003746841, cpktTxId=0000000000003746841)
482004 2015-08-08 15:10:46,306 INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Purging remote journals older than txid 2749022
482005 2015-08-08 15:10:46,306 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Purging logs older than 2749022                              
482006 2015-08-08 15:10:47,024 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1438136570797_9057
       /job_1438136570797_9057_1.jhist. BP-131542666-172.16.73.194-1433919468788 blk_1074205348_464553{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
       replicas=[ReplicaUnderConstruction[[DISK]DS-1759f904-d966-4ffb-8b6a-d061bfecfaac:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-b582cd84-5233-4e62-884
       1-13c5dfe4fa76:NORMAL|RBW]]}
active namenode是走到了
  public void purgeLogsOlderThan(long minTxIdToKeep)
      throws IOException {
    LOG.info("Purging logs older than " + minTxIdToKeep);
    File[] files = FileUtil.listFiles(sd.getCurrentDir());
    List<EditLogFile> editLogs =
      FileJournalManager.matchEditLogs(files);
    for (EditLogFile log : editLogs) {
      if (log.getFirstTxId() < minTxIdToKeep &&
          log.getLastTxId() < minTxIdToKeep) {
        purger.purgeLog(log);
      }
    }
  }

对比，现在的EDITS 和FSIMAGE数据，2749022是很久以前的数据了，是edits LOG的生存周期比较长，不知道有没有地方可以配置??
5650 -rw-rw-r-- 1 hadoop hadoop       42 8月   8 14:50 edits_0000000000003749720-0000000000003749721
5651 -rw-rw-r-- 1 hadoop hadoop    42854 8月   8 14:52 edits_0000000000003749722-0000000000003750042
5652 -rw-rw-r-- 1 hadoop hadoop    45064 8月   8 14:54 edits_0000000000003750043-0000000000003750372
5653 -rw-rw-r-- 1 hadoop hadoop     6320 8月   8 14:56 edits_0000000000003750373-0000000000003750413
5654 -rw-rw-r-- 1 hadoop hadoop       42 8月   8 14:58 edits_0000000000003750414-0000000000003750415
5655 -rw-rw-r-- 1 hadoop hadoop      175 8月   8 15:00 edits_0000000000003750416-0000000000003750418
5656 -rw-rw-r-- 1 hadoop hadoop    41878 8月   8 15:02 edits_0000000000003750419-0000000000003750733
5657 -rw-rw-r-- 1 hadoop hadoop    45966 8月   8 15:04 edits_0000000000003750734-0000000000003751068
5658 -rw-rw-r-- 1 hadoop hadoop     6333 8月   8 15:06 edits_0000000000003751069-0000000000003751109
5659 -rw-rw-r-- 1 hadoop hadoop       42 8月   8 15:08 edits_0000000000003751110-0000000000003751111
5660 -rw-rw-r-- 1 hadoop hadoop      175 8月   8 15:10 edits_0000000000003751112-0000000000003751114
5661 -rw-rw-r-- 1 hadoop hadoop    43086 8月   8 15:12 edits_0000000000003751115-0000000000003751438
5662 -rw-rw-r-- 1 hadoop hadoop    50815 8月   8 15:14 edits_0000000000003751439-0000000000003751802
5663 -rw-rw-r-- 1 hadoop hadoop      154 8月   8 15:16 edits_0000000000003751803-0000000000003751805
5664 -rw-rw-r-- 1 hadoop hadoop       42 8月   8 15:18 edits_0000000000003751806-0000000000003751807
5665 -rw-rw-r-- 1 hadoop hadoop      175 8月   8 15:20 edits_0000000000003751808-0000000000003751810
5666 -rw-rw-r-- 1 hadoop hadoop    43099 8月   8 15:22 edits_0000000000003751811-0000000000003752134
5667 -rw-rw-r-- 1 hadoop hadoop    50957 8月   8 15:24 edits_0000000000003752135-0000000000003752499
5668 -rw-rw-r-- 1 hadoop hadoop       42 8月   8 15:26 edits_0000000000003752500-0000000000003752501
5669 -rw-rw-r-- 1 hadoop hadoop       42 8月   8 15:28 edits_0000000000003752502-0000000000003752503
5670 -rw-rw-r-- 1 hadoop hadoop  1048576 8月   8 15:28 edits_inprogress_0000000000003752504
5671 -rw-rw-r-- 1 hadoop hadoop 16829185 8月   8 14:40 fsimage_0000000000003749021
5672 -rw-rw-r-- 1 hadoop hadoop       62 8月   8 14:40 fsimage_0000000000003749021.md5
5673 -rw-rw-r-- 1 hadoop hadoop 16842507 8月   8 15:10 fsimage_0000000000003751114
5674 -rw-rw-r-- 1 hadoop hadoop       62 8月   8 15:10 fsimage_0000000000003751114.md5
5675 -rw-rw-r-- 1 hadoop hadoop        8 8月   8 15:28 seen_txid
5676 -rw-rw-r-- 1 hadoop hadoop      205 8月   6 10:02 VERSION