分享

hbase master log中一直报出如下错误,如何修复

shihuai355 发表于 2016-4-25 14:45:56 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 5 12931
hbase master log中一直报出如下错误,请问谁知道有谁遇过这个问题,如何修复呢
我们用的cloudera 5.4,对应的hbase版本是1.0.0
2016-04-18 19:18:43,997 ERROR org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner: Exception while checking if files were valid, keeping them just in case.
com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: table_schema
    at com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:81)
    at com.google.protobuf.AbstractParser.checkMessageInitialized(AbstractParser.java:71)
    at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:217)
    at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:223)
    at com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
    at org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$SnapshotDataManifest.parseFrom(SnapshotProtos.java:4094)
    at org.apache.hadoop.hbase.snapshot.SnapshotManifest.readDataManifest(SnapshotManifest.java:518)
    at org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:358)
    at org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:122)
    at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:128)
    at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.getHFileNames(SnapshotReferenceUtil.java:357)
    at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.getHFileNames(SnapshotReferenceUtil.java:340)
    at org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner$1.filesUnderSnapshot(SnapshotHFileCleaner.java:85)
    at org.apache.hadoop.hbase.master.snapshot.SnapshotFileCache.getSnapshotsInProgress(SnapshotFileCache.java:303)
    at org.apache.hadoop.hbase.master.snapshot.SnapshotFileCache.getUnreferencedFiles(SnapshotFileCache.java:194)
    at org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner.getDeletableFiles(SnapshotHFileCleaner.java:62)
    at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteFiles(CleanerChore.java:233)
    at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:157)
    at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteDirectory(CleanerChore.java:180)
    at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:149)
    at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteDirectory(CleanerChore.java:180)
    at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:149)
    at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteDirectory(CleanerChore.java:180)
    at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:149)
    at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteDirectory(CleanerChore.java:180)
    at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:149)
    at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteDirectory(CleanerChore.java:180)
    at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:149)
    at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteDirectory(CleanerChore.java:180)
    at org.apache.hadoop.hbase.master.cleaner.CleanerChore.checkAndDeleteEntries(CleanerChore.java:149)
    at org.apache.hadoop.hbase.master.cleaner.CleanerChore.chore(CleanerChore.java:124)
    at org.apache.hadoop.hbase.Chore.run(Chore.java:87)
    at java.lang.Thread.run(Thread.java:745)

已有(5)人评论

跳转到指定楼层
leo_1989 发表于 2016-4-25 15:56:13
谷歌的protobuf有问题了吧。看看运行是否正常。
导致hbase快照的时候出问题了
回复

使用道具 举报

shihuai355 发表于 2016-4-25 16:28:17
如何查看protobuf有问题没,目前这个集群hadoop和hbase服务都是可以用的,在这个集群是通过hbase shell是可以做快照的并删除快照的,但是日志确一直持续报这个错,也不知道是哪个操作引起这个错误的
回复

使用道具 举报

s060403072 发表于 2016-4-25 17:04:48
[mw_shl_code=bash,true]protoc --version[/mw_shl_code]
回复

使用道具 举报

cherishWang 发表于 2016-6-11 01:00:49
遇到了相同的问题,请问解决了吗?能否共享下解决方法,谢谢
回复

使用道具 举报

cherishWang 发表于 2016-6-11 03:02:55

原因分析:
查看了原代码,错误发生在从SnapshotFileCache 中getName()的时候抛出来的IOException,这个Cache默认是每5分钟刷新一次。当Cache中无法getName的时候,就会报告这个错误。所以,导致错误的原因,可能是Cache被清理了,已经不存在对应的SnapshotFileCache,清理Snapshot的进程拿不到对应的HTable (table_schema)

我们的解决方法:
整个Hbase Service重启,让系统重新初始化SnapshotFileCache。然后,Hbase Snapshot Clean就可以正常清理了。

原代码:
SnapshotHFileCleaner.java

[mw_shl_code=java,true]/** Refresh cache, by default, every 5 minutes */
  private static final long DEFAULT_HFILE_CACHE_REFRESH_PERIOD = 300000;

  /** File cache for HFiles in the completed and currently running snapshots */
  private SnapshotFileCache cache;

  @Override
  public synchronized boolean isFileDeletable(Path filePath) {
    try {
      return !cache.contains(filePath.getName());
    } catch (IOException e) {
      LOG.error("Exception while checking if:" + filePath + " was valid, keeping it just in case.",
        e);
      return false;
    }
  }
[/mw_shl_code]
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条