图文精华

About云-梭伦科技»专题 › 交流区 › 技术交流 › HBase › Hbase导入大量数据，regionserver异常奔溃
分享

Hbase导入大量数据，regionserver异常奔溃

zkk2011 发表于 2016-11-9 17:53:10 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 5 34883
在向HBASE中导入数据（利用的是java api)，运行了差不多1分钟后，regionserver就会提示报警，压缩队列大小存在隐患，
  • GC 持续时间 存在隐患，过一会就会进程异常退出。求大神指点

回复

使用道具 举报

自定义广告语

已有(5)人评论

跳转到指定楼层
正序浏览
nextuser 发表于 2016-11-9 18:54:52
是不是内存溢出，具体还需要看日志
回复

使用道具 举报

nextuser 发表于 2016-11-9 18:58:41
是否超时的原因，设置下超时时间
在hdfs端和HBase端，给dfs.socket.timeout设置一个较大的值，比如300000（300秒）【注意两处设置的值要相等】


回复

使用道具 举报

easthome001 发表于 2016-11-9 19:55:05
调优下jvm试试
回复

使用道具 举报

zkk2011 发表于 2016-11-10 13:57:50
nextuser 发表于 2016-11-9 18:54
是不是内存溢出，具体还需要看日志

我查了日志  应该是memstore大于阈值导致的

11月 10, 中午12点14:36.122
WARN
org.apache.hadoop.hbase.regionserver.wal.MetricsWAL
regionserver/hadoop.node1/192.168.1.91:60020.append-
pool1-t1 took 2664 ms appending an edit to wal; len~=775.8 K
11月 10, 中午12点14:36.169
WARN
org.apache.hadoop.hbase.ipc.RpcServer
(responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.
ClientProtos$MultiRequest)","starttimems":1478751261440,"
responsesize":69386,"method":"Multi","processingtimems":
14703,"client":"192.168.20.82:47126","queuetimems":0,"class":"
HRegionServer"}
11月 10, 中午12点14:36.169
WARN
org.apache.hadoop.hbase.ipc.RpcServer
(responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.
ClientProtos$MultiRequest)","starttimems":1478751261584,"
responsesize":69386,"method":"Multi","processingtimems":
14548,"client":"192.168.20.82:47126","queuetimems":0,"
class":"HRegionServer"}
11月 10, 中午12点14:36.143
WARN
org.apache.hadoop.hbase.ipc.RpcServer
(responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.
ClientProtos$MultiRequest)","starttimems":1478751261283,"
responsesize":69386,"method":"Multi","processingtimems":
14859,"client":"192.168.20.82:47126","queuetimems":1,"class":"
HRegionServer"}
11月 10, 中午12点14:36.127
INFO
org.apache.hadoop.hbase.regionserver.wal.FSHLog
Slow sync cost: 2668 ms, current pipeline:
[DatanodeInfoWithStorage
[192.168.1.91:50010,DS-604714ab-3a68-4c32-a95f-98a380aa27a2,
DISK], DatanodeInfoWithStorage[192.168.1.94:50010,
DS-6aad11f0-81da-4ee0-8961-4588aa96a098,DISK],
DatanodeInfoWithStorage[192.168.1.200:50010,
11月 10, 中午12点14:37.212
WARN
org.apache.hadoop.hbase.ipc.RpcServer
(responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.
generated.ClientProtos$MultiRequest)","starttimems":
1478751263816,"responsesize":69386,"method":"Multi","
processingtimems":13394,"client":"192.168.20.82:47126",
"queuetimems":0,"class":"HRegionServer"}
11月 10, 中午12点14:37.213
WARN
org.apache.hadoop.hbase.ipc.RpcServer
(responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.
ClientProtos$MultiRequest)","starttimems":1478751263755,"
responsesize":69386,"method":"Multi","processingtimems":
13452,"client":"192.168.20.82:47126","queuetimems":203,
"class":"HRegionServer"}
11月 10, 中午12点14:37.213
WARN
org.apache.hadoop.hbase.ipc.RpcServer
(responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.
ClientProtos$MultiRequest)","starttimems":1478751263540,"
responsesize":69386,"method":"Multi","processingtimems":
13671,"client":"192.168.20.82:47126","queuetimems":1,"
class":"HRegionServer"}
11月 10, 中午12点14:38.299
WARN
org.apache.hadoop.hbase.regionserver.wal.MetricsWAL
regionserver/hadoop.node1/192.168.1.91:60020.
append-pool1-t1 took 1014 ms appending an edit to wal;
len~=775.8 K
11月 10, 中午12点14:38.311
WARN
org.apache.hadoop.hbase.ipc.RpcServer
(responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.
generated.ClientProtos$MultiRequest)","starttimems":
1478751264138,"responsesize":69386,"method":"
Multi","processingtimems":14162,"client":"
192.168.20.82:47126","queuetimems":7,"class":"HRegionServer"}
11月 10, 中午12点14:38.449
WARN
org.apache.hadoop.hbase.ipc.RpcServer
(responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.
ClientProtos$MultiRequest)","starttimems":1478751266904,"
responsesize":69386,"method":"Multi","processingtimems":11534,"
client":"192.168.20.82:47126","queuetimems":2,"class":"
HRegionServer"}
11月 10, 中午12点14:38.531
WARN
org.apache.hadoop.hbase.ipc.RpcServer
(responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.generated.
ClientProtos$MultiRequest)","starttimems":147875126695
3,"responsesize":69386,"method":"Multi","processingtime
ms":11578,"client":"192.168.20.82:47126","queuetimems"
:0,"class":"HRegionServer"}
11月 10, 中午12点14:39.406
WARN
org.apache.hadoop.hbase.ipc.RpcServer
(responseTooSlow): {"call":"Multi(org.apache.hadoop.hbase.protobuf.gene
rated.ClientProtos$MultiRequest)","starttimems":14
78751267032,"responsesize":69386,"method":"Multi",
"processingtimems":12374,"client":"192.168.20.82:471
26","queuetimems":18,"class":"HRegionServer"}
11月 10, 中午12点14:39.526
INFO
org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher
Flushed, sequenceid=53285, memsize=2.7 M, hasBloo
mFilter=true, into tmp file hdfs://master:8020/hbase/dat
a/default/Password/70c2
f8c8771ae0f392aaa5729b3dbf97/.tmp/c75175b76e60
4763b66362d21fc61fd7
11月 10, 中午12点14:39.550
INFO
org.apache.hadoop.hbase.regionserver.HStore
Added hdfs://master:8020/hbase/data/default/Password/70c
2f8c8771ae0f392aaa5729b3dbf97/Info/c75175b76
e604763b66362d21fc61fd7, entries=13556, seque
nceid=53285, filesize=1.0 M
11月 10, 中午12点14:39.551
INFO
org.apache.hadoop.hbase.regionserver.HRegion
Finished memstore flush of ~2.69 MB/2819648, curre
ntsize=0 B/0 for region Password,08f4a3dd,1478660
314484.70c2f8c8771ae0f392aaa5729b3dbf97. in 6113ms, s
equenceid=53285, compaction requested=false
11月 10, 中午12点14:39.551
INFO
org.apache.hadoop.hbase.regionserver.HRegion
Flushing 1/1 column families, memstore=2.64 MB
11月 10, 中午12点14:39.611
INFO
org.apache.hadoop.hbase.regionserver.HStore
Completed compaction of 3 file(s) in Info of Passwor
d,09ecdf6c,1478749477196.a747280d35688782988
d6aa617e23cd0. into d8db587434cc4829ac883d99b
99fc37f(size=21.9 M), total size for store is 267.7 M.
This selection was in queue
for 0sec, and took 25sec to execute.
11月 10, 中午12点14:39.611
INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread
Completed compaction: Request = regionName=P
assword,09ecdf6c,1478749477196.a747280d35688
782988d6aa617e23cd0., storeName=Info, fileCou
nt=3, fileSize=21.9 M, priority=6, time=71423208806
861; duration=25sec
11月 10, 中午12点14:39.615
INFO
org.apache.hadoop.hbase.regionserver.HRegion
Starting compaction on Info in region Password,09d9c37
91,1478750849933.669bb1f22ec0738305fbc2c241c7fa39.
11月 10, 中午12点14:39.615
INFO
org.apache.hadoop.hbase.regionserver.HStore
Starting compaction of 3 file(s) in Info of Password,09d9c
3791,1478750849933.669bb1f22ec0738305fbc2c241c7fa
39. into tmpdir=hdfs://master:8020/hbase/data/default/Passwor
d/669bb1f22ec0738305fbc2c241c7fa39/.tmp, totalSize=7.5 M
11月 10, 中午12点14:39.620
INFO
org.apache.hadoop.hbase.io.hfile.CacheConfig
blockCache=LruBlockCache{blockCount=54, currentSiz
e=1216848, freeSize=127836648, maxSize=1290534
96, heapSize=1216848, minSize=122600816, minFac
tor=0.95, multiSize=61300408, multiFactor=0.5, singl
eSize=30650204, singleFactor=0.25}, cacheDataOnRead
=true, cacheDataOnWrite=false, cacheIndexesOnWrite=fals
e, cacheBloomsOnWrite=false, cacheEvictOnClose=false, ca
cheDataCompressed=false, prefetchOnOpen=false
11月 10, 中午12点14:39.651
INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Blocking updates on hadoop.node1,60020,1478682678
689: the global memstore size 123.6 M is >= than bloc
king 123.1 M size
11月 10, 中午12点14:39.656
INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Blocking updates on hadoop.node1,60020,1478682
678689: the global memstore size 125.6 M is >= than b
locking 123.1 M size
11月 10, 中午12点14:39.656
INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Blocking updates on hadoop.node1,60020,1478682
678689: the global memstore size 125.6 M is >= tha
n blocking 123.1 M size
11月 10, 中午12点14:39.656
INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Blocking updates on hadoop.node1,60020,147868
2678689: the global memstore size 125.6 M is >= t
han blocking 123.1 M size
11月 10, 中午12点14:39.656
INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Blocking updates on hadoop.node1,60020,1478
682678689: the global memstore size 125.6 M is >
= than blocking 123.1 M size
11月 10, 中午12点14:39.749
INFO
org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher
Flushed, sequenceid=53692, memsize=2.6 M, hasBl
oomFilter=true, into tmp file hdfs://master:8020/hb
ase/data/default/Password/1
ba546d5d62e7af816c60f2ac6eac1ae/.tmp/cbeed6
8a0ab94f64a00aac6401f87db4
11月 10, 中午12点14:39.760
INFO
org.apache.hadoop.hbase.regionserver.HStore
Added hdfs://master:8020/hbase/data/default/Password/1
ba546d5d62e7af816c60f2ac6eac1ae/Info/cbeed68
a0ab94f64a00aac6401f87db4, entries=13309, sequen
ceid=53692, filesize=1019.1 K
11月 10, 中午12点14:39.761
INFO
org.apache.hadoop.hbase.regionserver.HRegion
Finished memstore flush of ~2.64 MB/2768272,
currentsize=0 B/0 for region Password,0830083b
79,1478662683916.1ba546d5d62e7af816c60f2ac
6eac1ae. in 210ms, sequenceid=53692, compaction r
equested=true
11月 10, 中午12点14:39.761
WARN
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Memstore is above high water mark and block 110ms
11月 10, 中午12点14:39.761
WARN
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Memstore is above high water mark and block 110ms
11月 10, 中午12点14:39.761
WARN
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Memstore is above high water mark and block 110ms
11月 10, 中午12点14:39.761
WARN
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Memstore is above high water mark and block 108ms
11月 10, 中午12点14:39.761
WARN
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Memstore is above high water mark and block 110ms
11月 10, 中午12点14:39.761
INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Flush of region Password,09f66de6b,147875116272
1.21441f15177c8b3566fcbd2068f44b99. due to glo
bal heap pressure. Total Memstore size=124.9 M, Reg
ion memstore size=2.5 M
11月 10, 中午12点14:39.761
INFO
org.apache.hadoop.hbase.regionserver.HRegion
Flushing 1/1 column families, memstore=2.47 MB
11月 10, 中午12点14:39.822
INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Blocking updates on hadoop.node1,60020,14786826
78689: the global memstore size 125.0 M is >= than
blocking 123.1 M size
11月 10, 中午12点14:39.900
INFO
org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher
Flushed, sequenceid=44817, memsize=63.6 M,
hasBloomFilter=true, into tmp file hdfs://master:8020
/hbase/data/default/Password/
28d4f85a657df464e81e6ca59f37b3ab/.tmp/a67e2a
4110c94b4a999ed4b0db0b74cf
11月 10, 中午12点14:39.908
INFO
org.apache.hadoop.hbase.regionserver.HStore
Added hdfs://master:8020/hbase/data/default/Password/
28d4f85a657df464e81e6ca59f37b3ab/Info/a67e2a
4110c94b4a999ed4b0db0b74cf, entries=320602,
sequenceid=44817, filesize=24.6 M
11月 10, 中午12点14:39.911
INFO
org.apache.hadoop.hbase.regionserver.HRegion
Finished memstore flush of ~63.60 MB/66685216,
currentsize=39.59 MB/41512640 for region Password,09fb37096,1478751162721.28d4f85a657
df464e81e6ca59f37b3ab. in 25977ms, sequenceid=44817, compaction requested=false
11月 10, 中午12点14:39.911
WARN
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Memstore is above high water mark and block 89ms
11月 10, 中午12点14:39.911
INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Unblocking updates for server hadoop.node1,60020,1478682678689
11月 10, 中午12点14:39.912
WARN
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Memstore is above high water mark and block 261ms
11月 10, 中午12点14:39.912
INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Unblocking updates for server hadoop.node1,60020,1478682678689
11月 10, 中午12点14:39.913
WARN
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Memstore is above high water mark and block 260ms
11月 10, 中午12点14:39.913
INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Unblocking updates for server hadoop.node1,60020,1478682678689
11月 10, 中午12点14:39.914
WARN
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Memstore is above high water mark and block 263ms
11月 10, 中午12点14:39.914
INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Unblocking updates for server hadoop.node1,60020,1478682678689
11月 10, 中午12点14:39.914
WARN
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Memstore is above high water mark and block 263ms
11月 10, 中午12点15:01.751
ERROR
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Cache flusher failed for entry org.apache.hadoop.hbase.regions
erver.MemStoreFlush
er$WakeupFlushThread@7217ab2bjava.lang.IllegalStateException        at com.google.common.base.Preconditions.checkState(Pr
econditions.java:133)        at org.apache.hadoop.hbas
e.regionserver.MemStoreFlu
sher.flushOneForGlobalPressure(MemStoreFlusher.java:191)        at org.apache.hadoop.hbase.regionserver.MemStoreFlu
sher.access$600(MemStoreFlusher.java:75)        at org.apache.hadoop.hbase.regionserver.MemStoreFlush
er$FlushHandler.run(MemStoreFlusher.java:244)        at
java.lang.Thread.run(Thread.java:745)
11月 10, 中午12点15:01.834
ERROR
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Cache flusher failed for entry org.apache.hadoop.hbase
.regionserver.MemStoreFlus
tateException        at com.google.common.base.Precond
itions.checkState(P
reconditions.java:133)        at org.apache.hadoop.hba
se.regionserver.MemStoreFlu
sher.flushOneForGlobalPressure(MemStoreFlusher.java:191)        at org.apache.hadoop.hbase.regionserver.MemStoreFlushe
r.access$600(MemStoreFlusher.java:75)        at org.a
pache.hadoop.hbase.regionserver.MemStoreFlush
er$FlushHandler.run(MemStoreFlusher.java:244)        a
t java.lang.Thread.run(Thread.java:745)
11月 10, 中午12点15:01.940
ERROR
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Cache flusher failed for entry org.apache.hadoop.hb
ase.regionserver.MemSto
ang.IllegalStateException        at com.google.com
mon.base.Preconditions.checkState(Preconditions.j
ava:133)        at org.apache.hadoop.hbase.regionserver.MemSto
reFlusher.flushOneForGlobalPressure(MemStoreFlus
her.java:191)        at org.apache.hadoop.hbase.regi
onserver.MemStoreFlusher.access$600(MemStoreF
lusher.java:75)        at org.apache.hadoop.hbase.reg
ionserver.MemStore
Flusher$FlushHandler.run(MemStoreFlusher.java:244)      
  at java.lang.Thread.run(Thread.java:745)
11月 10, 中午12点15:02.068
ERROR
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Cache flusher failed for entry org.apache.hadoop.hbase.regionserver.MemStoreFlush
teException        at com.google.common.base.Prec
onditions.checkState(Preconditions.java:133)        at or
g.apache.hadoop.hbase.regionserver.MemStore
Flusher.flushOneForGlobalPressure(MemStoreFlusher.java:191)   
     at org.apache.hadoop.hbase.regionserver.MemStoreFlu
sher.access$600(MemStoreFlusher.java:75)        at org.apac
he.hadoop.hbase.regionserver.MemStoreFlu
sher$FlushHandler.run(MemStoreFlusher.java:244)        at
java.lang.Thread.run(Thread.java:745)
11月 10, 中午12点15:02.069
ERROR
org.apache.hadoop.hbase.regionserver.MemStoreFlusher
Cache flusher failed for entry org.apache.hadoop.hbase.regionserver.MemStoreFlu
alStateException        at com.google.common.base.
Preconditions.checkState(Preconditions.java:133)        at org.apache.hadoop.hbase.regionserver.MemStoreFlu
sher.flushOneForGlobalPressure(MemStoreFlusher.java:191)        at org.apache.hadoop.hbase.regionserver.MemStoreFlus
her.access$600(MemStoreFlusher.java:75)        at org.apache.hadoop.hbase.regionserver.MemStoreFlu
sher$FlushHandler.run(MemStoreFlusher.java:244)        at java.lang.Thread.run(Thread.java:745)


请教这个该怎么解决？
回复

使用道具 举报

arsenduan 发表于 2016-11-10 15:31:43
首先保证内存是够的，并且尝试增大这下面两个参数
hbase.hregion.memstore.block.multiplier
hbase.hstore.blockingStoreFiles

回复

使用道具 举报

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

发表新帖
zkk2011

注册会员

关注

  • 1

    主题

  • 11

    帖子

  • 3

    粉丝

TA的主题
关闭

推荐上一条 /2 下一条

关于我们· 联系我们· 加入我们· 小黑屋· 合作伙伴   

Copyright © 2001-2024 About云-梭伦科技 Powered by Discuz! X3.4 Licensed Discuz Team.

简书 / Smrz 京ICP备2020039040号 Wxb 简书网举报电话：021-34700000