zookeeper中每天都有timeout的报错

我有5个zookeeper节点组成一个集群
23,24,25,26,27
在每一个节点上都有相同的报错
[mw_shl_code=applescript,true]2019-05-20 05:47:19,665 [myid:1] - INFO  [CommitProcessor:1:ZooKeeperServer@687] - Established session 0x16aa72e41ea0169 with negotiated timeout 30000 for client /10.3.87.23:57553
2019-05-20 06:32:57,958 [myid:1] - INFO  [CommitProcessor:1:ZooKeeperServer@687] - Established session 0x16aa72e41ea016a with negotiated timeout 30000 for client /10.3.87.23:34304
2019-05-20 06:38:01,912 [myid:1] - INFO  [CommitProcessor:1:ZooKeeperServer@687] - Established session 0x16aa72e41ea016b with negotiated timeout 30000 for client /10.3.87.23:19989
2019-05-20 06:43:05,789 [myid:1] - INFO  [CommitProcessor:1:ZooKeeperServer@687] - Established session 0x16aa72e41ea016c with negotiated timeout 30000 for client /10.3.87.23:56055
2019-05-20 07:18:35,354 [myid:1] - INFO  [CommitProcessor:1:ZooKeeperServer@687] - Established session 0x16aa72e41ea016d with negotiated timeout 30000 for client /10.3.87.23:45158
2019-05-20 07:49:00,969 [myid:1] - INFO  [CommitProcessor:1:ZooKeeperServer@687] - Established session 0x16aa72e41ea016e with negotiated timeout 30000 for client /10.3.87.23:45014
2019-05-20 08:09:17,947 [myid:1] - INFO  [CommitProcessor:1:ZooKeeperServer@687] - Established session 0x16aa72e41ea016f with negotiated timeout 30000 for client /10.3.87.23:14279
2019-05-20 08:34:16,129 [myid:1] - INFO  [CommitProcessor:1:ZooKeeperServer@687] - Established session 0x16aa72e41ea0170 with negotiated timeout 30000 for client /10.3.87.23:57366
2019-05-20 08:34:16,153 [myid:1] - INFO  [CommitProcessor:1:ZooKeeperServer@687] - Established session 0x16aa72e41ea0171 with negotiated timeout 30000 for client /10.3.87.23:57384
2019-05-20 08:34:16,164 [myid:1] - INFO  [CommitProcessor:1:ZooKeeperServer@687] - Established session 0x16aa72e41ea0172 with negotiated timeout 30000 for client /10.3.87.23:57392
2019-05-20 08:34:16,179 [myid:1] - INFO  [CommitProcessor:1:ZooKeeperServer@687] - Established session 0x16aa72e41ea0173 with negotiated timeout 30000 for client /10.3.87.23:57400
2019-05-20 08:34:16,183 [myid:1] - INFO  [CommitProcessor:1:ZooKeeperServer@687] - Established session 0x16aa72e41ea0174 with negotiated timeout 30000 for client /10.3.87.23:57402
2019-05-20 08:39:45,161 [myid:1] - INFO  [CommitProcessor:1:ZooKeeperServer@687] - Established session 0x16aa72e41ea0175 with negotiated timeout 30000 for client /10.3.87.23:56001
[/mw_shl_code]
不知道在那里查找原因
而且多数指向23这台机器，少数指向别的机器

是否安装ntp，看下时间是否同步。

date看了一下时间应该是同步的

ighack 发表于 2019-5-20 11:54
date看了一下时间应该是同步的

一个端口有问题说明端口的问题，多个端口有问题，要么是进程挂掉了，要么是网络造成的。看看防火墙，hosts，或则配置文件等因素造成的网络问题。
还有趣那台机器看下错误日志，肯定有错误产生。

程序没有挂掉。我用supervisord管理zookeeper。我ping 23值是正常的。kafka的日志中也没发现连不上zookeeper的错误
关键23这台机器也报这样的错。感觉不是网络的问题。我让网络部的人去查。也没发现网有什么问题
Established session 0x16aa72e41ea0175 with negotiated timeout 30000 for client /10.3.87.23:56001
配制文件我用的是IP没有用机器名配制。hosts我也看了。都是对的

这个报错也不是一直报。时间上也没有一个规律



顺便说一下。我的kafka也是在这5台机器上建的集群。运行一段时间就报错。我重启一下这5台机器就好了。不知道是什么原因

[mw_shl_code=applescript,true]2019-05-20 11:26:03,562 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@373] - Exception causing close of session 0x0 due to java.io.IOException: Len error 11957
258562019-05-20 11:26:05,563 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@373] - Exception causing close of session 0x0 due to java.io.IOException: Len error 11957
258562019-05-20 11:26:08,564 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@373] - Exception causing close of session 0x0 due to java.io.IOException: Len error 11957
258562019-05-20 14:43:24,554 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@373] - Exception causing close of session 0x0 due to java.io.IOException: Len error 11957
258562019-05-20 14:43:25,554 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@373] - Exception causing close of session 0x0 due to java.io.IOException: Len error 11957
258562019-05-20 14:43:27,555 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@373] - Exception causing close of session 0x0 due to java.io.IOException: Len error 11957
258562019-05-20 14:43:30,556 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@373] - Exception causing close of session 0x0 due to java.io.IOException: Len error 11957
25856root@jzyc1.com:/tmp#
[/mw_shl_code]
我发现23上有这样的错。但和报超时对不上啊。超时是每天都有。这个只有今天下午发现了几个错
其他的机器上没有报这个错。网上有说是3.4.6的BUG但我用的是zookeeper-3.4.10

ighack 发表于 2019-5-20 15:26
[mw_shl_code=applescript,true]2019-05-20 11:26:03,562 [myid:1] - WARN  [NIOServerCxn.Factory:0.0.0.0 ...

可以在升级下，或则配置下面属性试试：
zkCli.sh Configuration: -Djute.maxbuffer=41943040

zkServer.sh Configuration:  -Djute.maxbuffer=41943040

记得备份，做好还原。并且记得生效

java.io.IOException: Len error 11957这个错误只有昨天出现了这几条
主要还是超时的问题不知道是什么原因

我现在重启了一下zookeeper
2019-05-20 08:39:45,161 [myid:1] - INFO [CommitProcessor:1:ZooKeeperServer@687] - Established session 0x16aa72e41ea0175 with negotiated timeout 30000 for client /10.3.87.23:56001
这样的警告少了很多。但感觉我的zookeeper集群的连接数不平衡，有的节点14连接，有的节点4个连接，有的只有1个连接
kafka中的
WARN Attempting to send response via channel for which there is no open connection
这样的警告还是有很多
kafka中的警告是不是和我调参有关
num.network.threads=8
num.io.threads=16

WARN Attempting to send response via channel for which there is no open connection
这个一天有120多个