本帖最后由 xioaxu790 于 2014-9-6 18:13 编辑 问题导读1、在非安全环境下运行Hadoop，要配置什么？ 3、如何管理 NodeManagers的健康？ 准备 安装 在非安全环境下运行Hadoop Site 配置 配置Hadoop守护进程的环境 守护进程 环境变量 NameNode HADOOP_NAMENODE_OPTS DataNode HADOOP_DATANODE_OPTS Secondary NameNode HADOOP_SECONDARYNAMENODE_OPTS ResourceManager YARN_RESOURCEMANAGER_OPTS NodeManager YARN_NODEMANAGER_OPTS WebAppProxy YARN_PROXYSERVER_OPTS Map Reduce Job History Server HADOOP_JOB_HISTORYSERVER_OPTS
export HADOOP_NAMENODE_OPTS="-XX:+UseParallelGC ${HADOOP_NAMENODE_OPTS}" 复制代码 守护进程 环境变量 ResourceManager YARN_RESOURCEMANAGER_HEAPSIZE NodeManager YARN_NODEMANAGER_HEAPSIZE WebAppProxy YARN_PROXYSERVER_HEAPSIZE Map Reduce Job History Server HADOOP_JOB_HISTORYSERVER_HEAPSIZE
参数 值 说明 fs.defaultFS NameNode URI hdfs://host:port/ io.file.buffer.size 131072 在序列文件中使用的读/写缓冲大小。
conf/hdfs-site.xml 参数 值 说明 dfs.namenode.name.dir NameNode在本地文件系统上的路径，永久地存储了命名空间和传输日志。 dfs.namenode.hosts /dfs.namenode.hosts.exclude 允许/非允许 DataNodes列表. 若有必要，用这个文件控制允许的datanode列表。 dfs.blocksize 268435456 在大型文件系统中HDFS 块大小为256MB 。 dfs.namenode.handler.count 100 更多的 NameNode 服务器线程处理来自DataNode大量的RPCs。
参数
用于一个 DataNode存放块的本地文件系统上逗号分隔的路径列表
如果这是一个逗号分隔的词典，那么数据就被存在所有的命名的词典中，通常在不同设备上。
conf/yarn-site.xml 参数 值 说明 yarn.acl.enable true /false 允许 ACLs默认是 false. yarn.admin.acl Admin ACL 在集群上设置administrator。 ACLs are of for comma-separated-usersspacecomma-separated-groups.默认是指定值为*表示任何人。特别的是空格表示皆无权限。 yarn.log-aggregation-enable false 配置允许日志聚合与否。
ResourceManager 的配置: 参数
yarn.resourcemanager.address
ResourceManager host:port 用于客户端提交任务。
yarn.resourcemanager.scheduler.address
ResourceManager host:port 用于应用管理者向 Scheduler获取资源
yarn.resourcemanager.resource-tracker.address
ResourceManager host:port 用于 NodeManagers.
yarn.resourcemanager.admin.address
ResourceManager host:port用于管理命令
yarn.resourcemanager.webapp.address
ResourceManager 浏览器界面 host:port.
yarn.resourcemanager.scheduler.class
ResourceManager Scheduler class.
CapacityScheduler (推荐 ), FairScheduler(亦推荐 ), 或 FifoScheduler
yarn.scheduler.minimum-allocation-mb
在 Resource Manager上为每个容器分配的最小内存。
yarn.scheduler.maximum-allocation-mb
在 Resource Manager上为每个容器分配的最大内存。
yarn.resourcemanager.nodes.include-path /yarn.resourcemanager.nodes.exclude-path
如有必要，用这些文件控制允许的 NodeManager列表。
NodeManager 配置: 参数
yarn.nodemanager.resource.memory-mb
Resource i.e. available physical memory, in MB, for givenNodeManager
Defines total available resources on the NodeManager to be made available to running containers
yarn.nodemanager.vmem-pmem-ratio
Maximum ratio by which virtual memory usage of tasks may exceed physical memory
The virtual memory usage of each task may exceed its physical memory limit by this ratio. The total amount of virtual memory used by tasks on the NodeManager may exceed its physical memory usage by this ratio.
yarn.nodemanager.local-dirs
Comma-separated list of paths on the local filesystem where intermediate data is written.
Multiple paths help spread disk i/o.
yarn.nodemanager.log-dirs
Comma-separated list of paths on the local filesystem where logs are written.
Multiple paths help spread disk i/o.
yarn.nodemanager.log.retain-seconds
Default time (in seconds) to retain log files on the NodeManager Only applicable if log-aggregation is disabled.
yarn.nodemanager.remote-app-log-dir
HDFS directory where the application logs are moved on application completion. Need to set appropriate permissions. Only applicable if log-aggregation is enabled.
yarn.nodemanager.remote-app-log-dir-suffix
Suffix appended to the remote log dir. Logs will be aggregated to ${yarn.nodemanager.remote-app-log-dir}/${user}/${thisParam} Only applicable if log-aggregation is enabled.
yarn.nodemanager.aux-services
Shuffle service that needs to be set for Map Reduce applications.
History Server 的配置 (需要移到其他地方): 参数
yarn.log-aggregation.retain-seconds
How long to keep aggregation logs before deleting them. -1 disables. Be careful, set this too small and you will spam the name node.
yarn.log-aggregation.retain-check-interval-seconds
Time between checks for aggregated log retention. If set to 0 or a negative value then the value is computed as one-tenth of the aggregated log retention time. Be careful, set this too small and you will spam the name node.
conf/mapred-site.xml Parameter
Execution framework set to Hadoop YARN.
Larger resource limit for maps.
Larger heap-size for child jvms of maps.
mapreduce.reduce.memory.mb
Larger resource limit for reduces.
mapreduce.reduce.java.opts
Larger heap-size for child jvms of reduces.
mapreduce.task.io.sort.mb
Higher memory-limit while sorting data for efficiency.
mapreduce.task.io.sort.factor
More streams merged at once while sorting files.
mapreduce.reduce.shuffle.parallelcopies
Higher number of parallel copies run by reduces to fetch outputs from very large number of maps.
Parameter Value Notes mapreduce.jobhistory.address MapReduce JobHistory Server host:port Default port is 10020. mapreduce.jobhistory.webapp.address MapReduce JobHistory Server Web UIhost:port Default port is 19888. mapreduce.jobhistory.intermediate-done-dir /mr-history/tmp Directory where history files are written by MapReduce jobs. mapreduce.jobhistory.done-dir /mr-history/done Directory where history files are managed by the MR JobHistory Server.
Hadoop 机架感知 管理 NodeManagers的健康 Parameter Value Notes yarn.nodemanager.health-checker.script.path Node health script Script to check for node's health status. yarn.nodemanager.health-checker.script.opts Node health script options Options for script to check for node's health status. yarn.nodemanager.health-checker.script.interval-ms Node health script interval Time interval for running health script. yarn.nodemanager.health-checker.script.timeout-ms Node health script timeout interval Timeout for health script execution.
Slaves 文件 日志 操作 Hadoop 集群 $ $HADOOP_PREFIX/bin/hdfs namenode -format <cluster_name> 复制代码 $ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode 复制代码 $ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode 复制代码 $ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode 复制代码 $ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager 复制代码 $ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager
$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start proxyserver --config $HADOOP_CONF_DIR 复制代码 $ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR 复制代码 $ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start proxyserver --config $HADOOP_CONF_DIR 复制代码 $ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver --config $HADOOP_CONF_DIR 复制代码 $ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode 复制代码 $ $HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode 复制代码 $ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager 复制代码 $ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop nodemanager 复制代码 $ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh stop proxyserver --config $HADOOP_CONF_DIR 复制代码 $ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh stop historyserver --config $HADOOP_CONF_DIR 复制代码 在安全模式下运行 Hadoop User:Group
NameNode, Secondary NameNode, Checkpoint Node, Backup Node, DataNode
ResourceManager, NodeManager
MapReduce JobHistory Server
Filesystem
yarn.nodemanager.local-dirs
yarn.nodemanager.log-dirs
conf/container-executor.cfg
yarn.nodemanager.remote-app-log-dir
mapreduce.jobhistory.intermediate-done-dir
mapreduce.jobhistory.done-dir
安全认证 keytab 文件 HDFS $ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/nn.service.keytab
Keytab name: FILE:/etc/security/keytab/nn.service.keytab
KVNO Timestamp Principal
4 07/18/11 21:08:09 nn/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 nn/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 nn/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) 复制代码 $ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/sn.service.keytab
Keytab name: FILE:/etc/security/keytab/sn.service.keytab
KVNO Timestamp Principal
4 07/18/11 21:08:09 sn/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 sn/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 sn/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) 复制代码 $ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/dn.service.keytab
Keytab name: FILE:/etc/security/keytab/dn.service.keytab
KVNO Timestamp Principal
4 07/18/11 21:08:09 dn/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 dn/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 dn/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) 复制代码 YARN $ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/rm.service.keytab
Keytab name: FILE:/etc/security/keytab/rm.service.keytab
KVNO Timestamp Principal
4 07/18/11 21:08:09 rm/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 rm/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 rm/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) 复制代码 $ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/nm.service.keytab
Keytab name: FILE:/etc/security/keytab/nm.service.keytab
KVNO Timestamp Principal
4 07/18/11 21:08:09 nm/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 nm/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 nm/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5) 复制代码 $ /usr/kerberos/bin/klist -e -k -t /etc/security/keytab/jhs.service.keytab
Keytab name: FILE:/etc/security/keytab/jhs.service.keytab
KVNO Timestamp Principal
4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 jhs/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-256 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (AES-128 CTS mode with 96-bit SHA-1 HMAC)
4 07/18/11 21:08:09 host/full.qualified.domain.name@REALM.TLD (ArcFour with HMAC/md5)
复制代码 WEB界面 守护进程
MapReduce JobHistory Server
/2 
京ICP备2020039040号
简书网举报电话：021-34700000
提升卡
置顶卡
沉默卡
喧嚣卡
变色卡
千斤顶
显身卡
