Cloudera Manager5 在线bin安装遇到 无法检测到 Agent 发出的检测信号 总结

不期而遇 发表于 2016-10-28 11:56
我试过楼主的方法，但是依然提示无法检测agent信号，以下是我的cloudera-scm-agent的日志信息：[28/Oct/201 ...

可能是因为httpd服务有问题。

service httpd restart
重启找下原因
特别检测下hosts

执行service httpd restart 之后并且还重装过系统
但是还是依然上次同样的错误：
ARQUET_HOME': '/usr/lib/parquet', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'CMF_PACKAGE_DIR': '/usr/lib64/cmf/service', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'PATH': '/sbin:/usr/sbin:/bin:/usr/bin', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'KEYTRUSTEE_SERVER_HOME': '/usr/lib/keytrustee-server', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
[31/Oct/2016 14:49:10 +0000] 3607 MainThread agent        INFO     To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
[31/Oct/2016 14:49:10 +0000] 3607 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/process
[31/Oct/2016 14:49:10 +0000] 3607 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor
[31/Oct/2016 14:49:10 +0000] 3607 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/flood
[31/Oct/2016 14:49:10 +0000] 3607 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include
[31/Oct/2016 14:49:10 +0000] 3607 MainThread agent        INFO     Supervisor version: 3.0
[31/Oct/2016 14:49:10 +0000] 3607 MainThread agent        INFO     Connecting to previous supervisor: agent-29909-1477885709.
[31/Oct/2016 14:49:11 +0000] 3607 MainThread status_server INFO     Using maximum impala profile bundle size of 1073741824 bytes.
[31/Oct/2016 14:49:11 +0000] 3607 MainThread status_server INFO     Using maximum stacks log bundle size of 1073741824 bytes.
[31/Oct/2016 14:49:11 +0000] 3607 MainThread _cplogging   INFO     [31/Oct/2016:14:49:11] ENGINE Bus STARTING
[31/Oct/2016 14:49:11 +0000] 3607 MainThread _cplogging   INFO     [31/Oct/2016:14:49:11] ENGINE Started monitor thread '_TimeoutMonitor'.
[31/Oct/2016 14:49:11 +0000] 3607 HTTPServer Thread-2 _cplogging   ERROR    [31/Oct/2016:14:49:11] ENGINE Error in HTTP server: shutting down
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py", line 187, in _start_http_thread
    self.httpserver.start()
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py", line 1825, in start
    raise socket.error(msg)
error: No socket could be created on ('wanfin3', 9000) -- [Errno 99] Cannot assign requested address

[31/Oct/2016 14:49:11 +0000] 3607 HTTPServer Thread-2 _cplogging   INFO     [31/Oct/2016:14:49:11] ENGINE Bus STOPPING
[31/Oct/2016 14:49:11 +0000] 3607 HTTPServer Thread-2 _cplogging   INFO     [31/Oct/2016:14:49:11] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('wanfin3', 9000)) already shut down
[31/Oct/2016 14:49:11 +0000] 3607 HTTPServer Thread-2 _cplogging   INFO     [31/Oct/2016:14:49:11] ENGINE Stopped thread '_TimeoutMonitor'.
[31/Oct/2016 14:49:11 +0000] 3607 HTTPServer Thread-2 _cplogging   INFO     [31/Oct/2016:14:49:11] ENGINE Bus STOPPED
[31/Oct/2016 14:49:11 +0000] 3607 HTTPServer Thread-2 _cplogging   INFO     [31/Oct/2016:14:49:11] ENGINE Bus EXITING
[31/Oct/2016 14:49:11 +0000] 3607 HTTPServer Thread-2 _cplogging   INFO     [31/Oct/2016:14:49:11] ENGINE Bus EXITED
我的hosts文件如下：
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.94 wanfin1
192.168.0.95 wanfin2
192.168.0.96 wanfin3
  执行python -c 'import socket; print socket.getfqdn(), socket.gethostbyname(socket.getfqdn())'
显示ip和主机名是一一对应的。真是搞不懂了，现在怀疑是不是这三台虚拟机是openstack在物理机的基础上虚拟出来的机器，不能安装cdh

pig2 发表于 2016-10-29 19:05
可能是因为httpd服务有问题。

service httpd restart

我试了楼主的方法，还是不行，我这里的机器是通过openstack在物理服务器虚拟出来好多虚拟机然后用其中3台做一个测试环境。会不会和这有关，搞不懂之前没有出现过这个问题的。

不期而遇 发表于 2016-10-31 15:27
我试了楼主的方法，还是不行，我这里的机器是通过openstack在物理服务器虚拟出来好多虚拟机然后用其中3台 ...

应该没有关系

把ip6禁用掉
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

@einhep 首选非常感谢你能回答我的问题，但是试了你的方法还是没用。cloudera-scm-agent.log里面的报错
'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY HBASE_TABLE_DISABLED HBASE_TABLE_ENABLED JOBTRACKER_IN_STANDBY_MODE YARN_RM_IN_STANDBY_MODE', 'KEYTRUSTEE_KP_HOME': '/usr/share/keytrustee-keyprovider', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'KEYTRUSTEE_SERVER_HOME': '/usr/lib/keytrustee-server', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_KMS_HOME': '/usr/lib/hadoop-kms', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_SQOOP_HOME': '/usr/lib/sqoop', 'WEBHCAT_DEFAULT_XML': '/etc/hive-webhcat/conf.dist/webhcat-default.xml', 'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HUE_HOME': '/usr/lib/hue', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SENTRY_HOME': '/usr/lib/sentry', 'CDH_CRUNCH_HOME': '/usr/lib/crunch', 'CDH_LLAMA_HOME': '/usr/lib/llama/', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HCAT_HOME': '/usr/lib/hive-hcatalog', 'CDH_KAFKA_HOME': '/usr/lib/kafka', 'CDH_SPARK_HOME': '/usr/lib/spark', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_FLUME_HOME': '/usr/lib/flume-ng'}
[31/Oct/2016 16:07:19 +0000] 1444 MainThread agent        INFO     To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
[31/Oct/2016 16:07:19 +0000] 1444 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/process
[31/Oct/2016 16:07:19 +0000] 1444 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor
[31/Oct/2016 16:07:19 +0000] 1444 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/flood
[31/Oct/2016 16:07:19 +0000] 1444 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include
[31/Oct/2016 16:07:19 +0000] 1444 MainThread agent        ERROR    Failed to connect to previous supervisor.
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.2-py2.6.egg/cmf/agent.py", line 2036, in find_or_start_supervisor
    self.configure_supervisor_clients()
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.7.2-py2.6.egg/cmf/agent.py", line 2220, in configure_supervisor_clients
    supervisor_options.realize(args=["-c", os.path.join(self.supervisor_dir, "supervisord.conf")])
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/supervisor-3.0-py2.6.egg/supervisor/options.py", line 1578, in realize
    Options.realize(self, *arg, **kw)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/supervisor-3.0-py2.6.egg/supervisor/options.py", line 312, in realize
    self.process_config()
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/supervisor-3.0-py2.6.egg/supervisor/options.py", line 320, in process_config
    self.process_config_file(do_usage)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/supervisor-3.0-py2.6.egg/supervisor/options.py", line 355, in process_config_file
    self.usage(str(msg))
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/supervisor-3.0-py2.6.egg/supervisor/options.py", line 143, in usage
    self.exit(2)
SystemExit: 2
[31/Oct/2016 16:07:19 +0000] 1444 MainThread tmpfs        INFO     Successfully mounted tmpfs at /var/run/cloudera-scm-agent/process
[31/Oct/2016 16:07:20 +0000] 1444 MainThread agent        INFO     Trying to connect to newly launched supervisor (Attempt 1)
[31/Oct/2016 16:07:20 +0000] 1444 MainThread agent        INFO     Supervisor version: 3.0
[31/Oct/2016 16:07:20 +0000] 1444 MainThread agent        INFO     Successfully connected to supervisor
[31/Oct/2016 16:07:20 +0000] 1444 MainThread status_server INFO     Using maximum impala profile bundle size of 1073741824 bytes.
[31/Oct/2016 16:07:20 +0000] 1444 MainThread status_server INFO     Using maximum stacks log bundle size of 1073741824 bytes.
[31/Oct/2016 16:07:20 +0000] 1444 MainThread _cplogging   INFO     [31/Oct/2016:16:07:20] ENGINE Bus STARTING
[31/Oct/2016 16:07:20 +0000] 1444 MainThread _cplogging   INFO     [31/Oct/2016:16:07:20] ENGINE Started monitor thread '_TimeoutMonitor'.
[31/Oct/2016 16:07:20 +0000] 1444 HTTPServer Thread-2 _cplogging   ERROR    [31/Oct/2016:16:07:20] ENGINE Error in HTTP server: shutting down
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/process/servers.py", line 187, in _start_http_thread
    self.httpserver.start()
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/CherryPy-3.2.2-py2.6.egg/cherrypy/wsgiserver/wsgiserver2.py", line 1825, in start
    raise socket.error(msg)
error: No socket could be created on ('wanfin3', 9000) -- [Errno 99] Cannot assign requested address

[31/Oct/2016 16:07:20 +0000] 1444 HTTPServer Thread-2 _cplogging   INFO     [31/Oct/2016:16:07:20] ENGINE Bus STOPPING
[31/Oct/2016 16:07:20 +0000] 1444 HTTPServer Thread-2 _cplogging   INFO     [31/Oct/2016:16:07:20] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('wanfin3', 9000)) already shut down
[31/Oct/2016 16:07:20 +0000] 1444 HTTPServer Thread-2 _cplogging   INFO     [31/Oct/2016:16:07:20] ENGINE Stopped thread '_TimeoutMonitor'.
[31/Oct/2016 16:07:20 +0000] 1444 HTTPServer Thread-2 _cplogging   INFO     [31/Oct/2016:16:07:20] ENGINE Bus STOPPED
[31/Oct/2016 16:07:20 +0000] 1444 HTTPServer Thread-2 _cplogging   INFO     [31/Oct/2016:16:07:20] ENGINE Bus EXITING
[31/Oct/2016 16:07:20 +0000] 1444 HTTPServer Thread-2 _cplogging   INFO     [31/Oct/2016:16:07:20] ENGINE Bus EXITED
其中有一台服务器的日志是这样的，其他两台日志还之前报错一样。

Last login: Mon Oct 31 16:26:00 2016 from 192.168.xx.xx每次登陆的ip不一致 会不会和这个有关系啊

[root@wanfin3 ~]# service cloudera-scm-agent status
cloudera-scm-agent (pid  2689) is running...
[root@wanfin3 ~]# service cloudera-scm-agent status
cloudera-scm-agent dead but pid file exists

不期而遇 发表于 2016-10-31 16:40
Last login: Mon Oct 31 16:26:00 2016 from 192.168.xx.xx每次 ...

这个是用vpn每次连接登录的时候会出现ip每次不一致的情况。

我也是一样的问题，而且我也找不到那两个文件在哪里、、、急等

@TataSong那个我也没有但是已经装好了