启动 Cloudera Management Service 报错
启动Service Monitorsupervisor: couldn't chdir to /opt/cm-5.14.2/run/cloudera-scm-agent/process/32-cloudera-mgmt-SERVICEMONITOR: EACCESsupervisor: child process was not spawned
server 日志：
2019-04-03 11:46:32,251 WARN 1268797474@scm-web-34:com.cloudera.server.web.cmf.HealthReportHelper: (31 skipped) Error getting health report from Service Monitor.
org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88)
at com.sun.proxy.$Proxy139.getAvroHealthReports(Unknown Source)
at com.cloudera.cmf.protocol.firehose.nozzle.TimeoutNozzleIPC.getAvroHealthReports(TimeoutNozzleIPC.java:127)
at com.cloudera.cmon.NozzleIPCWrapper.getHealthReports(NozzleIPCWrapper.java:599)
at com.cloudera.server.web.cmf.HealthReportHelper$GetHealthReportCallable.call(HealthReportHelper.java:502)
at com.cloudera.server.web.cmf.HealthReportHelper.getHealthReport(HealthReportHelper.java:350)
at com.cloudera.server.web.cmf.HealthCheckController.instanceStatusHealthCheckJSON(HealthCheckController.java:287)
at sun.reflect.GeneratedMethodAccessor1597.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:436)
at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:424)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:574)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78)
at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309)
at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:71)
at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58)
at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72)
at org.apache.avro.ipc.Requestor.request(Requestor.java:147)
at org.apache.avro.ipc.Requestor.request(Requestor.java:101)
at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:72)
... 75 more
agent 日志：
28211 MainThread agent INFO Triggering supervisord update.
28211 MainThread process INFO Begin audit plugin refresh
28211 MainThread navigator_plugin INFO Scheduling a refresh for Audit Plugin for cloudera-mgmt-SERVICEMONITOR with pipelines []
28211 MainThread process INFO Begin metadata plugin refresh
28211 MainThread navigator_plugin INFO Scheduling a refresh for Metadata Plugin for cloudera-mgmt-SERVICEMONITOR with pipelines []
28211 MainThread __init__ INFO Instantiating generic monitor for service MGMT and role SERVICEMONITOR
28211 MainThread process INFO Begin monitor refresh.
28211 MainThread abstract_monitor INFO Refreshing GenericMonitor MGMT-SERVICEMONITOR for None
28211 MainThread __init__ INFO New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x3168650>,)
28211 MainThread process INFO Daemon refresh complete for process 32-cloudera-mgmt-SERVICEMONITOR.
28211 CP Server Thread-4 _cplogging INFO 172.26.136.230 - - "GET /download_log?path=%2Fvar%2Flog%2Fcloudera-scm-firehose%2Fmgmt-cmf-mgmt-SERVICEMONITOR-KF-CFT-Hadoop01.log.out&onlyTail=true&compress=false HTTP/1.1" 404 796 "" "Java/1.8.0_201"
28211 Metadata-Plugin navigator_plugin INFO Refreshing Metadata Plugin for cloudera-mgmt-SERVICEMONITOR with pipelines []
28211 Metadata-Plugin navigator_plugin_pipeline INFO Stopping Navigator Plugin Pipeline '' for cloudera-mgmt-SERVICEMONITOR (log dir: None)
28211 Audit-Plugin navigator_plugin INFO Refreshing Audit Plugin for cloudera-mgmt-SERVICEMONITOR with pipelines []
28211 Audit-Plugin navigator_plugin_pipeline INFO Stopping Navigator Plugin Pipeline '' for cloudera-mgmt-SERVICEMONITOR (log dir: None)
28211 CP Server Thread-5 _cplogging INFO 172.26.136.230 - - "GET /download_log?path=%2Fvar%2Flog%2Fcloudera-scm-firehose%2Fmgmt-cmf-mgmt-SERVICEMONITOR-KF-CFT-Hadoop01.log.out&onlyTail=true&compress=false HTTP/1.1" 404 796 "" "Java/1.8.0_201"
28211 MainThread process INFO Updating process: False {u'running': (True, False), u'run_generation': (1, 2)}
28211 MainThread process INFO Deactivating process 32-cloudera-mgmt-SERVICEMONITOR
28211 Audit-Plugin navigator_plugin INFO stopping Audit Plugin for cloudera-mgmt-SERVICEMONITOR with pipelines []
28211 Metadata-Plugin navigator_plugin INFO stopping Metadata Plugin for cloudera-mgmt-SERVICEMONITOR with pipelines []
28211 Audit-Plugin navigator_plugin_pipeline INFO Stopping Navigator Plugin Pipeline '' for cloudera-mgmt-SERVICEMONITOR (log dir: None)
28211 Metadata-Plugin navigator_plugin_pipeline INFO Stopping Navigator Plugin Pipeline '' for cloudera-mgmt-SERVICEMONITOR (log dir: None)
28211 MonitorDaemon-Scheduler __init__ INFO Monitor expired: ('GenericMonitor MGMT-SERVICEMONITOR for mgmt-SERVICEMONITOR-bd3fcecc94a00ef083a2a3956ecf6209',)
上面有两处有价值的错误信息：
EACCESsupervisor: child process was not spawned
这一处错误太过广泛，意思是不能生成子进程，但是不能生成子进程的原因太多了
org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused (Connection refused)
这一处的错误是链接错误。也就是说上面错误的原因是因为这里被链接拒绝造成的，也就是说这里成为了关键。但是这个链接拒绝又有很多原因。比如端口被暂用，防火墙未关闭，ntp未同步时间等等，各种原因都可能造成。所以楼主还需要更详细的信息
yaojiank 发表于 2019-4-3 17:56
上面有两处有价值的错误信息：
EACCESsupervisor: child process was not spawned
agent :
30031 MainThread process INFO First time to activate the process .
30031 MainThread agent INFO Created /opt/cm-5.14.2/run/cloudera-scm-agent/process/70-cloudera-mgmt-SERVICEMONITOR
30031 MainThread agent INFO Chowning /opt/cm-5.14.2/run/cloudera-scm-agent/process/70-cloudera-mgmt-SERVICEMONITOR to cloudera-scm (987) cloudera-scm (982)
30031 MainThread agent INFO Chmod'ing /opt/cm-5.14.2/run/cloudera-scm-agent/process/70-cloudera-mgmt-SERVICEMONITOR to 0751
30031 MainThread agent INFO Created /opt/cm-5.14.2/run/cloudera-scm-agent/process/70-cloudera-mgmt-SERVICEMONITOR/logs
30031 MainThread agent INFO Chowning /opt/cm-5.14.2/run/cloudera-scm-agent/process/70-cloudera-mgmt-SERVICEMONITOR/logs to cloudera-scm (987) cloudera-scm (982)
30031 MainThread agent INFO Chmod'ing /opt/cm-5.14.2/run/cloudera-scm-agent/process/70-cloudera-mgmt-SERVICEMONITOR/logs to 0751
30031 MainThread process INFO Refreshing process files: None
30031 MainThread parcel INFO prepare_environment begin: {u'CDH': u'5.14.2-1.cdh5.14.2.p0.3'}, [], []
30031 MainThread parcel INFO Service does not request any parcels
30031 MainThread __init__ INFO Extracted 12 files and 0 dirs to /opt/cm-5.14.2/run/cloudera-scm-agent/process/70-cloudera-mgmt-SERVICEMONITOR.
30031 MainThread process INFO Evaluating resource: {u'io': None, u'named_cpu': None, u'tcp_listen': None, u'dynamic': True, u'rlimits': None, u'install': None, u'file': None, u'memory': None, u'directory': {u'path': u'/var/log/cloudera-scm-firehose/stacks', u'bytes_free_warning_threshhold_bytes': 0, u'group': u'cloudera-scm', u'user': u'cloudera-scm', u'mode': 493}, u'cpu': None, u'contents': None}
30031 MainThread os_ops WARNINGExpected mode 0755 for /var/log/cloudera-scm-firehose/stacks but was 0775
30031 MainThread process INFO reading limits: {u'limit_memlock': None, u'limit_fds': None}
30031 MainThread process INFO Launching process. one-off False, command mgmt/mgmt.sh, args
30031 MainThread agent INFO Triggering supervisord update.
30031 MainThread process INFO Begin audit plugin refresh
30031 MainThread navigator_plugin INFO Scheduling a refresh for Audit Plugin for cloudera-mgmt-SERVICEMONITOR with pipelines []
30031 MainThread process INFO Begin metadata plugin refresh
30031 MainThread navigator_plugin INFO Scheduling a refresh for Metadata Plugin for cloudera-mgmt-SERVICEMONITOR with pipelines []
30031 MainThread __init__ INFO Instantiating generic monitor for service MGMT and role SERVICEMONITOR
30031 MainThread process INFO Begin monitor refresh.
30031 MainThread abstract_monitor INFO Refreshing GenericMonitor MGMT-SERVICEMONITOR for None
30031 MainThread __init__ INFO New monitor: (<cmf.monitor.generic.GenericMonitor object at 0x2fdab50>,)
30031 MainThread process INFO Daemon refresh complete for process 70-cloudera-mgmt-SERVICEMONITOR.
30031 Metadata-Plugin navigator_plugin INFO Refreshing Metadata Plugin for cloudera-mgmt-SERVICEMONITOR with pipelines []
30031 Metadata-Plugin navigator_plugin_pipeline INFO Stopping Navigator Plugin Pipeline '' for cloudera-mgmt-SERVICEMONITOR (log dir: None)
30031 Audit-Plugin navigator_plugin INFO Refreshing Audit Plugin for cloudera-mgmt-SERVICEMONITOR with pipelines []
30031 Audit-Plugin navigator_plugin_pipeline INFO Stopping Navigator Plugin Pipeline '' for cloudera-mgmt-SERVICEMONITOR (log dir: None)
30031 MainThread process INFO Updating process: False {u'running': (True, False), u'run_generation': (1, 2)}
30031 MainThread process INFO Deactivating process 70-cloudera-mgmt-SERVICEMONITOR
30031 Metadata-Plugin navigator_plugin INFO stopping Metadata Plugin for cloudera-mgmt-SERVICEMONITOR with pipelines []
30031 Metadata-Plugin navigator_plugin_pipeline INFO Stopping Navigator Plugin Pipeline '' for cloudera-mgmt-SERVICEMONITOR (log dir: None)
30031 Audit-Plugin navigator_plugin INFO stopping Audit Plugin for cloudera-mgmt-SERVICEMONITOR with pipelines []
30031 Audit-Plugin navigator_plugin_pipeline INFO Stopping Navigator Plugin Pipeline '' for cloudera-mgmt-SERVICEMONITOR (log dir: None)
30031 MonitorDaemon-Scheduler __init__ INFO Monitor expired: ('GenericMonitor MGMT-SERVICEMONITOR for mgmt-SERVICEMONITOR-9087569f37dce8f46af56b79189cf761',)
学东 发表于 2019-4-3 22:07
agent :
30031 MainThread process INFO First time to activ ...
需要找到error
可以通过下面命令过滤日志中的错误。
cat 日志|grep ERROR
这个问题上周搞定了，同事卸载重装的、然后chmod 777 -R /opt 这一类问题就没有了。
重装卸载一定要卸干净。
学东 发表于 2019-4-9 10:19
这个问题上周搞定了，同事卸载重装的、然后chmod 777 -R /opt 这一类问题就没有了。
重装卸载一定要卸干净 ...
楼主说的对。
对于cloudera的安装，只要一步步按照步骤，一般不会出问题的。
如果是安装中出现异常，其实还是比较难搞的。
最好最粗暴的办法就是卸载干净重新安装。
如果想详细知道错误等，那就是十分耗时间了，而且可能还解决不掉。
页:
[1]