立即注册 登录
About云-梭伦科技 返回首页

496252649的个人空间 https://aboutyun.com/?10256 [收藏] [复制] [分享] [RSS]

日志

cdh5 hadoop 配置/etc/hadoop/conf/yarn-site.xml

已有 2234 次阅读2015-1-3 14:35 |个人分类:hadoop之路

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value></value>
    <description>默认值是“mapreduce.shuffle”,但如果在hadoop-2.2 中继续使用这个值,NodeManager 会启动失败改为“mapreduce_shuffle”。</description>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    <description>为了能够运行MapReduce程序,需要让各个NodeManager在启动时加载shuffle server,shuffle server实际上是Jetty/Netty Server,Reduce Task通过该server从各个NodeManager上远程拷贝Map Task产生的中间结果。上面增加的两个配置均用于指定shuffle serve。如果YARN集群有多个节点,你还要配置yarn.resourcemanager.address等参数???</description>
  </property>

  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
    <description>是否启用日志聚集功能。</description>
  </property>

  <property>
    <description>List of directories to store localized files in.</description>
    <name>yarn.nodemanager.local-dirs</name>
    <value>file:///var/lib/hadoop-yarn/cache/${user.name}/nm-local-dir</value>
  </property>

  <property>
    <description>Where to store container logs.</description>
    <name>yarn.nodemanager.log-dirs</name>
    <value>file:///var/log/hadoop-yarn/containers</value>
  </property>

  <property>
    <description>Where to aggregate logs to.</description>
    <name>yarn.nodemanager.remote-app-log-dir</name>
    <value>hdfs://var/log/hadoop-yarn/apps</value>
  </property>
  <property>
    <name>yarn.acl.enable</name>
    <value>true</value>
    <description>启用ACL的?默认为false。</description>
  </property>
  <property>
    <name>yarn.admin.acl</name>
    <value>*</value>
    <description>访问控制列表,在群集上设置管理员。 ACL是为逗号分隔usersspacecomma分隔组。默认为*特殊值这意味着任何人。只是空间的特殊值意味着没有人进入</description>
  </property>
  <property>
    <name>yarn.resourcemanager.address</name>
    <value>m001:8032</value>
    <description>ResourceManager 对客户端暴露的地址。客户端通过该地址向RM提交应用程序,杀死应用程序等。</description>
  </property>
  <property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>m001:8033</value>
    <description>ResourceManager 对管理员暴露的访问地址。管理员通过该地址向RM发送管理命令等。</description>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>m001:8030</value>
    <description>ResourceManager 对ApplicationMaster暴露的访问地址。ApplicationMaster通过该地址向RM申请资源、释放资源等。</description>
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>m001:8031</value>
    <description>ResourceManager 对NodeManager暴露的地址.。NodeManager通过该地址向RM汇报心跳,领取任务等。</description>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>m001:8088</value>
    <description>ResourceManager对外web ui地址。用户可通过该地址在浏览器中查看集群各类信息</description>
  </property>
  <property>
    <name>yarn.resourcemanager.client.thread-count</name>
    <value>50</value>
    <description>handler数量</description>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.client.thread-count</name>
    <value>50</value>
    <description>处理来自ApplicationMaster的RPC请求的Handler数目</description>
  </property>
  <property>
    <name>yarn.resourcemanager.admin.client.thread-count</name>
    <value>1</value>
    <description>线程用于处理应用程序管理器请求的数量。</description>
  </property>
  <property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>1024</value>
    <description>每个块的最低配置要求在RM,MBs。内存请求低于这不会生效,并将配置至少指定的值。</description>
  </property>
  <property>
    <name>yarn.scheduler.increment-allocation-mb</name>
    <value>512</value>
    <description>内存规整化单位,默认是1024,这意味着,如果一个Container请求资源是1.5GB,则将被调度器规整化为ceiling(1.5 GB / 1GB) * 1G=2GB。</description>
  </property>
  <property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>8192</value>
    <description>最大可申请内存量,默认是8096</description>
  </property>
  <property>
    <name>yarn.scheduler.minimum-allocation-vcores</name>
    <value>1</value>
    <description>最小可申请CPU数,默认是1</description>
  </property>
  <property>
    <name>yarn.scheduler.increment-allocation-vcores</name>
    <value>1</value>
    <description>虚拟CPU规整化单位,默认是1,含义与内存规整化单位类似</description>
  </property>
  <property>
    <name>yarn.scheduler.maximum-allocation-vcores</name>
    <value>20</value>
    <description>最大可申请CPU数,默认是4</description>
  </property>
  <property>
    <name>yarn.resourcemanager.amliveliness-monitor.interval-ms</name>
    <value>1000</value>
    <description>RM检查心跳</description>
  </property>
  <property>
    <name>yarn.am.liveness-monitor.expiry-interval-ms</name>
    <value>600000</value>
    <description>到期时间间隔为应用程序主报告</description>
  </property>
  <property>
    <name>yarn.resourcemanager.am.max-retries</name>
    <value>1</value>
    <description>The maximum number of application master retries.</description>
  </property>
  <property>
    <name>yarn.resourcemanager.container.liveness-monitor.interval-ms</name>
    <value>600000</value>
    <description>How often to check that containers are still alive. </description>
  </property>
  <property>
    <name>yarn.resourcemanager.nm.liveness-monitor.interval-ms</name>
    <value>1000</value>
    <description>How often to check that node managers are still alive.</description>
  </property>
  <property>
    <name>yarn.nm.liveness-monitor.expiry-interval-ms</name>
    <value>600000</value>
    <description>How long to wait until a node manager is considered dead.</description>
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.client.thread-count</name>
    <value>50</value>
    <description>Number of threads to handle resource tracker calls.</description>
  </property>
  <property>
    <name>yarn.application.classpath</name>
    <value>$HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*</value>
    <description>CLASSPATH for YARN applications. A comma-separated list of CLASSPATH entries</description>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
    <description>The class to use as the resource scheduler.</description>
  </property>
  <property>
    <name>yarn.scheduler.fair.user-as-default-queue</name>
    <value>true</value>
    <description>当应用程序未指定队列名时,是否指定用户名作为应用程序所在的队列名。如果设置为false或者未设置,所有未知队列的应用程序将被提交到default队列>中,默认值为true。</description>
  </property>
  <property>
    <name>yarn.scheduler.fair.preemption</name>
    <value>false</value>
    <description>是否启用抢占机制,默认值是false</description>
  </property>
  <property>
    <name>yarn.scheduler.fair.sizebasedweight</name>
    <value>false</value>
    <description>在一个队列内部分配资源时,默认情况下,采用公平轮询的方法将资源分配各各个应用程序,而该参数则提供了另外一种资源分配方式:按照应用程序资源>需求数目分配资源,即需求资源数量越多,分配的资源越多。默认情况下,该参数值为false。</description>
  </property>
  <property>
    <name>yarn.scheduler.fair.assignmultiple</name>
    <value>false</value>
    <description>是否允许多个容器分配在一个心跳。默认为false。</description>
  </property>
  <property>
    <name>yarn.resourcemanager.max-completed-applications</name>
    <value>10000</value>
    <description>完成应用程序的最大数量RM</description>
  </property>
</configuration>

路过

雷人

握手

鲜花

鸡蛋

评论 (0 个评论)

facelist doodle 涂鸦板

您需要登录后才可以评论 登录 | 立即注册

关闭

推荐上一条 /2 下一条