hbase meta表修复方式总结
问题导读
1.hbase hbck -fixMeta的作用是什么？
2.如何重新将hbase meta表分给regionserver？
3.出现region的hole该如何修复?
static/image/hrline/4.gif
meta表修复一
查看hbasemeta情况
hbase hbck
1.重新修复hbase meta表
hbase hbck -fixMeta
2.重新将hbase meta表分给regionserver
hbase hbck -fixAssignments
static/image/hrline/2.gif
转meta，表手动删除表
因为集群硬盘紧俏，绝对对原来的表加上COMPRESSION=>LZO属性。但是创建表，长时间没有反馈。决定drop掉这张表，但是始终drop失败。重启集群，hbase 60010界面显示有region transaction。为创建失败的表region，在PENDING_OPEN和CLOSED之间跳。describe 表失败， enable表失败，disable表失败，从60010界面查看表失败。很蛋疼。
后决定强制删除当前表。google了一下，找到这篇文章，文章大部分都是对的，但是最后一步存在问题.原文中命令为：
delete 'TrojanInfo','TrojanInfo,,1361433390076.2636b5a2b3d3d08f23d2af9582f29bd8.','info:server'
当时就觉得有有问题，没有涉及.META.表，如何更新META信息？
尝试两次删除后是始终报错，确定应该是有问题，为了以防万一，google一下更新META信息的操作，将命令改为
delete '.META.','TrojanInfo,,1361433390076.2636b5a2b3d3d08f23d2af9582f29bd8.','info:server'
命令成功执行。
重启集群后，transction仍然存在，分析应该是meta表没有更新的问题，对meta表做一次major_compact，重启集群，成功。不再有报错。
下面是对原文的拷贝：
强制删除表：
1、强制删除该表在hdfs上的所有文件(路径根据实际情况而定）：
view plaincopy
./hadoop fs -rmr /hbase/TrojanInfo
2、删除该表在HBase系统表.META.中的记录：
A、首先从.META.中查询出表 TrojanInfo在.META.中的rowkey，这可以通过scan '.META.'，然后手动筛选；
B、然后删除该rowkey下的3个字段（假设查询出的rowkey为TrojanInfo,,1361433390076.2636b5a2b3d3d08f23d2af9582f29bd8.）
view plaincopy
delete 'TrojanInfo','TrojanInfo,,1361433390076.2636b5a2b3d3d08f23d2af9582f29bd8.','info:server'
delete 'TrojanInfo','TrojanInfo,,1361433390076.2636b5a2b3d3d08f23d2af9582f29bd8.','info:serverstartcode'
delete 'TrojanInfo','TrojanInfo,,1361433390076.2636b5a2b3d3d08f23d2af9582f29bd8.','info:reg
static/image/hrline/2.gif
meta表修复三
一、故障原因
IP为10.191.135.3的服务器在2013年8月1日出现服务器重新启动的情况，导致此台服务器上的所有服务均停止。从而造成NTP服务停止。当NTP服务停止后，导致HBase集群中大部分机器时钟和主机时间不一致，造成regionserver服务中止。并在重新启动后，出现region的hole。需要对数据进行重新修复，以正常提供插入数据的服务。
二、恢复方式
1、集群50个regionserver，宕掉服务41个，namenode所在机器10.191.135.3不明重启（原因查找中）导致本机上的namenode、zookeeper、时间同步服务器服务挂掉。
2、重启hbase服务时，没能成功stop剩余的9个regionserver服务，进行了人为kill进程，
3、在hdfs上移走了hlog（避免启动时split log花费过多时间影响服务），然后重启hbase。发现10.191.135.30机器上的时间与时间同步服务器10.191.135.3不同步。手工同步后重启成功。hbase可以正常提供查询服务。
4、运行mapreduce put数据。抛出异常，数据无法正常插入；
5、执行/opt/hbase/bin/hbase hbck -fixAssignments，尝试重新分配region。结果显示hbase有空洞，即region之间数据不连续了；
6、通过上述操作可以定位是在regionserver服务宕掉的后重启的过程中丢了数据。需要进行空洞修复。然而hbase hbck命令总是只显示三条空洞。
7、通过编写的regionTest.jar工具进行进一步检测出空洞所在的regionname然后停掉hbase，进而进行region合并修复空洞;
8、合并的merge 操作需要先去.META.表里读取该region的信息，由于.META.表也在regionserver宕机过程中受到损坏，所以部分region的.META.信息没有，merge操作时就抛出空指针异常。因此只能将hdfs这些region进行移除，然后通过regionTest.jar 检测新的空洞所在的regionname，进行合并操作修复空洞；
9、关于region重叠，即regionname存在.META.表内，但是在hdfs上被错误的移出，并进行了region合并。这种情况下需要通过regionTest.jar检测重叠的regionname然后手动去.META.表删除，.META.表修改之后需要flush；
10、最后再次执行 hbase hbck 命令，hbase 所有表status ok。
三、相关命令及页面报错信息
1.手工同步时间命令 service ntpd stop ntpdate -d 192.168.1.20 service ntpd start
2.org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed 2 actions: WrongRegionException: 2 times, servers with issues: datanode10:60020, at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchCallback(HConnectionManager.java:1641) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1409) at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:949) at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:826) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:801) at org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:123) at org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:84) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:533) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:88) at o
3.13/08/01 18:30:02 DEBUG util.HBaseFsck: There are 22093 region info entries ERROR: There is a hole in the region chain between +8615923208069cmnet201303072132166264580 and +861592321.You need to create a new .regioninfo and region dir in hdfs to plug the hole. ERROR: There is a hole in the region chain between +8618375993383cmwap20130512235639430 and +8618375998629cmnet201305040821436779670.You need to create a new .regioninfo and region dir in hdfs to plug the hole. ERROR: There is a hole in the region chain between +8618725888080cmnet201212271719506311400 and +8618725889786cmnet201302131646431671140.You need to create a new .regioninfo and region dir in hdfs to plug the hole. ERROR: Found inconsistency in table cqgprs Summary: -ROOT- is okay. Number of regions: 1 Deployed on:datanode14,60020,1375330955915 .META. is okay. Number of regions: 1 Deployed on:datanode21,60020,1375330955825 cqgprs is okay. Number of regions: 22057 Deployed on:datanode1,60020,1375330955761 datanode10,60020,1375330955748 datanode11,60020,1375330955736 datanode12,60020,1375330955993 datanode13,60020,1375330955951 datanode14,60020,1375330955915 datanode15,60020,1375330955882 datanode16,60020,1375330955892 datanode17,60020,1375330955864 datanode18,60020,1375330955703 datanode19,60020,1375330955910 datanode2,60020,1375330955751 datanode20,60020,1375330955849 datanode21,60020,1375330955825 datanode22,60020,1375334479752 datanode23,60020,1375330955835 datanode24,60020,1375330955932 datanode25,60020,1375330955856 datanode26,60020,1375330955807 datanode27,60020,1375330955882 datanode28,60020,1375330955785 datanode29,60020,1375330955799 datanode3,60020,1375330955778 datanode30,60020,1375330955748 datanode31,60020,1375330955877 datanode32,60020,1375330955763 datanode33,60020,1375330955755 datanode34,60020,1375330955713 datanode35,60020,1375330955768 datanode36,60020,1375330955896 datanode37,60020,1375330955884 datanode38,60020,1375330955918 datanode39,60020,1375330955881 datanode4,60020,1375330955826 datanode40,60020,1375330955770 datanode41,60020,1375330955824 datanode42,60020,1375449245386 datanode43,60020,1375330955880 datanode44,60020,1375330955902 datanode45,60020,1375330955881 datanode46,60020,1375330955841 datanode47,60020,1375330955790 datanode48,60020,1375330955848 datanode49,60020,1375330955849 datanode5,60020,1375330955880 datanode50,60020,1375330955802 datanode6,60020,1375330955753 datanode7,60020,1375330955890 datanode8,60020,1375330955967 datanode9,60020,1375330955948 test1 is okay. Number of regions: 1 Deployed on:datanode43,60020,1375330955880 test2 is okay. Number of regions: 1 Deployed on:datanode21,60020,1375330955825 35 inconsistencies detected. Status: INCONSISTENT
4.hadoop jar regionTest.jar com.region.RegionReaderMain /hbase/cqgprs 检测cqgprs表里的空洞所在的regionname。
5.================================== first endKey = +8615808059207cmnet201307102326567966800 second startKey = +8615808058578cmnet201212251545557984830 first regionNmae = cqgprs,+8615808058578cmnet201212251545557984830,1375241186209.0f8266ad7ac45be1fa7233e8ea7aeef9. second regionNmae = cqgprs,+8615808058578cmnet201212251545557984830,1362778571889.3552d3db8166f421047525d6be39c22e. ================================== first endKey = +8615808060140cmnet201303051801355846850 second startKey = +8615808059207cmnet201307102326567966800 first regionNmae = cqgprs,+8615808058578cmnet201212251545557984830,1362778571889.3552d3db8166f421047525d6be39c22e. second regionNmae = cqgprs,+8615808059207cmnet201307102326567966800,1375241186209.09d489d3df513bc79bab09cec36d2bb4. ==================================
6.Usage: bin/hbase org.apache.hadoop.hbase.util.Merge [-Dfs.default.name=hdfs://nn:port] <table-name> <region-1> <region-2> ./hbase org.apache.hadoop.hbase.util.Merge -Dfs.defaultFS=hdfs://bdpha cqgprs cqgprs,+8615213741567cmnet201305251243290802280,1369877465524.3c13b460fae388b1b1a70650b66c5039. cqgprs,+8615213745577cmnet201302141725552206710,1369534940433.5de80f59071555029ac42287033a4863. &
7.13/08/01 22:24:02 WARN util.HBaseFsck: Naming new problem group: +8618225125357cmnet201212290358070667800 ERROR: (regions cqgprs,+8618225123516cmnet201304131404096748520,1375363774655.b3cf5cc752f4427a4e699270dff9839e. and cqgprs,+8618225125357cmnet201212290358070667800,1364421610707.7f7038bfbe2c0df0998a529686a3e1aa.) There is an overlap in the region chain. 13/08/01 22:24:02 WARN util.HBaseFsck: reached end of problem group: +8618225127504cmnet201302182135452100210 13/08/01 22:24:02 WARN util.HBaseFsck: Naming new problem group: +8618285642723cmnet201302031921019768070 ERROR: (regions cqgprs,+8618285277826cmnet201306170027424674330,1375363962312.9d1e93b22cec90fd75361fa65b1d20d2. and cqgprs,+8618285642723cmnet201302031921019768070,1360873307626.f631cd8c6acc5e711e651d13536abe94.) There is an overlap in the region chain. 13/08/01 22:24:02 WARN util.HBaseFsck: reached end of problem group: +8618286275556cmnet201212270713444340110 13/08/01 22:24:02 WARN util.HBaseFsck: Naming new problem group: +8618323968833cmnet201306010239025175240 ERROR: (regions cqgprs,+8618323967956cmnet201306091923411365860,1375364143678.665dba6a14ebc9971422b39e079b00ae. and cqgprs,+8618323968833cmnet201306010239025175240,1372821719159.6d2fecc1b3f9049bbca83d84231eb365.) There is an overlap in the region chain. 13/08/01 22:24:02 WARN util.HBaseFsck: reached end of problem group: +8618323992353cmnet201306012336364819810 ERROR: There is a hole in the region chain between +8618375993383cmwap20130512235639430 and +8618375998629cmnet201305040821436779670.You need to create a new .regioninfo and region dir in hdfs to plug the hole. 13/08/01 22:24:02 WARN util.HBaseFsck: Naming new problem group: +8618723686187cmnet201301191433522129820 ERROR: (regions cqgprs,+8618723683087cmnet201301300708363045080,1375364411992.4ee5787217c1da4895d95b3b92b8e3a2. and cqgprs,+8618723686187cmnet201301191433522129820,1362003066106.70b48899cc753a0036f11bb27d2194f9.) There is an overlap in the region chain. 13/08/01 22:24:02 WARN util.HBaseFsck: reached end of problem group: +8618723689138cmnet201301051742388948390 13/08/01 22:24:02 WARN util.HBaseFsck: Naming new problem group: +8618723711808cmnet201301031139206225900 ERROR: (regions cqgprs,+8618723710003cmnet201301250809235976320,1375364586329.40eed10648c9a43e3d5ce64e9d63fe00. and cqgprs,+8618723711808cmnet201301031139206225900,1361216401798.ebc442e02f5e784bce373538e06dd232.) There is an overlap in the region chain. 13/08/01 22:24:02 WARN util.HBaseFsck: reached end of problem group: +8618723714626cmnet201302122009459491970 ERROR: There is a hole in the region chain between +8618725888080cmnet201212271719506311400 and +8618725889786cmnet201302131646431671140.You need to create a new .regioninfo and region dir in hdfs to plug the hole.
8.delete '.META.','regionname','info:serverstartcode'
delete '.META.','regionname','info:regionserver'
delete '.META.','regionname','info:regioninfo'
9. flush '.META.' major_compact '.META.'
虽然帖子很久了，想知道修复2中regionTest.jar是自己写的吧？
页:
[1]