hadoop2.X如何将namenode与SecondaryNameNode分开配置

问题导读

1.如何将namenode与SecondaryNameNode分开？

2.SecondaryNameNode单独配置，需要修改那些配置文件？


3.masters文件的作用是什么？





我们这里假设你已经安装配置了hadoop2.2，至于如何配置可以参考，hadoop2.2完全分布式最新高可靠安装文档。
在这个基础上，我们对配置文件做一些修改：
1.增加masters文件

sudo vi masters
复制代码

这里面放什么内容还是比较关键的，这里我们指定slave1节点上运行SecondaryNameNode。

注意：如果你想单独配置一台机器，那么在这个文件里面，填写这个节点的ip地址或则是hostname，如果是多台，则在masters里面写上多个，一行一个，我们这里指定一个

slave1
复制代码

2.修改hdfs-site.xml
在下面文件中增加如下内容：（记得下面亦可写成ip地址，这里为了理解方便，写的是hostname）

<property>
<name>dfs.http.address</name>
<value>master:50070</value>
<description>
The address and the base port where the dfs namenode web ui will listen on.
If the port is 0 then the server will start on a free port.
</description>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>slave1:50090</value>
</property>

3.修改core-site.xml文件

<property>
  <name>fs.checkpoint.period</name>
  <value>3600</value>
  <description>The number of seconds between two periodic checkpoints.
  </description>
</property>
<property>
  <name>fs.checkpoint.size</name>
  <value>67108864</value>
</property>
复制代码

上面修改完毕，相应的节点也做同样的修改





下面我们开始启动节点：

start-dfs.sh
复制代码

输出如下内容：

Starting namenodes on [master]
master: starting namenode, logging to /usr/hadoop/logs/hadoop-aboutyun-namenode-master.out
slave2: starting datanode, logging to /usr/hadoop/logs/hadoop-aboutyun-datanode-slave2.out
slave1: starting datanode, logging to /usr/hadoop/logs/hadoop-aboutyun-datanode-slave1.out
Starting secondary namenodes [slave1]
slave1: starting secondarynamenode, logging to /usr/hadoop/logs/hadoop-aboutyun-secondarynamenode-slave1.out
复制代码

然后查看节点：
（1）master节点：

aboutyun@master:/usr/hadoop/etc/hadoop$ jps
5994 NameNode
6201 Jps
复制代码

（2）slave1节点

aboutyun@slave1:/usr/hadoop/etc/hadoop$ jps
5199 SecondaryNameNode
5015 DataNode
5291 Jps
复制代码

（3）slave2节点

aboutyun@slave2:/usr/hadoop/etc/hadoop$ jps
3628 DataNode
3696 Jps
复制代码

停止节点：

master: stopping namenode
slave1: stopping datanode
slave2: stopping datanode
Stopping secondary namenodes [slave1]
slave1: stopping secondarynamenode
复制代码

补充
Secondarynamenode的单独启动和停止

启动：

   bin/hadoop-daemons.sh --config conf/ --hosts masters start secondarynamenode

停止：

   bin/hadoop-daemons.sh --config conf/ --hosts masters stop secondarynamenode

请问hadoop2.4里secondarynamenode总是在执行start-dfs.sh后自动起来，也没配置secondarynamenode，怎么不自动启动secondarynamenode呢？

临云 发表于 2014-6-25 14:48
请问hadoop2.4里secondarynamenode总是在执行start-dfs.sh后自动起来，也没配置secondarynamenode，怎么不 ...

hadoop的启动命令：
start-dfs.sh
一旦执行这个脚本，它就会启动master，slave，然后secondarynamenode。
也就是说start-dfs.sh对它们进行了定义，你可以修改里面的脚本。


首先进入hadoop的sbin目录，

aboutyun@master:/usr/hadoop/sbin$ sudo nano start-dfs.sh
#!/usr/bin/env bash

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.


# Start hadoop dfs daemons.
# Optinally upgrade or rollback dfs state.
# Run this on master node.

usage="Usage: start-dfs.sh [-upgrade|-rollback] [other options such as -clusterId]"

bin=`dirname "${BASH_SOURCE-$0}"`
bin=`cd "$bin"; pwd`

DEFAULT_LIBEXEC_DIR="$bin"/../libexec
HADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}
. $HADOOP_LIBEXEC_DIR/hdfs-config.sh

# get arguments
if [ $# -ge 1 ]; then
nameStartOpt="$1"
shift
case "$nameStartOpt" in
(-upgrade)
;;
(-rollback)
dataStartOpt="$nameStartOpt"
;;
(*)
echo $usage
exit 1
;;
esac
fi

#Add other possible options
nameStartOpt="$nameStartOpt $@"

#---------------------------------------------------------
# namenodes

NAMENODES=$($HADOOP_PREFIX/bin/hdfs getconf -namenodes)

echo "Starting namenodes on [$NAMENODES]"

"$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \
--config "$HADOOP_CONF_DIR" \
--hostnames "$NAMENODES" \
--script "$bin/hdfs" start namenode $nameStartOpt

#---------------------------------------------------------
# datanodes (using default slaves file)

if [ -n "$HADOOP_SECURE_DN_USER" ]; then
echo \
"Attempting to start secure cluster, skipping datanodes. " \
"Run start-secure-dns.sh as root to complete startup."
else
"$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \
--config "$HADOOP_CONF_DIR" \
--script "$bin/hdfs" start datanode $dataStartOpt
fi

#---------------------------------------------------------
# secondary namenodes (if any)

SECONDARY_NAMENODES=$($HADOOP_PREFIX/bin/hdfs getconf -secondarynamenodes 2>/dev/null)

if [ -n "$SECONDARY_NAMENODES" ]; then
echo "Starting secondary namenodes [$SECONDARY_NAMENODES]"

"$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \
--config "$HADOOP_CONF_DIR" \
--hostnames "$SECONDARY_NAMENODES" \
--script "$bin/hdfs" start secondarynamenode
fi

#---------------------------------------------------------
# quorumjournal nodes (if any)

SHARED_EDITS_DIR=$($HADOOP_PREFIX/bin/hdfs getconf -confKey dfs.namenode.shared.edits.dir 2>&-)

case "$SHARED_EDITS_DIR" in
qjournal://*)
JOURNAL_NODES=$(echo "$SHARED_EDITS_DIR" | sed 's,qjournal://\([^/]*\)/.*,\1,g; s/;/ /g; s/:[0-9]*//g')
echo "Starting journal nodes [$JOURNAL_NODES]"
"$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \
--config "$HADOOP_CONF_DIR" \
--hostnames "$JOURNAL_NODES" \
--script "$bin/hdfs" start journalnode ;;
esac

#---------------------------------------------------------
# ZK Failover controllers, if auto-HA is enabled
AUTOHA_ENABLED=$($HADOOP_PREFIX/bin/hdfs getconf -confKey dfs.ha.automatic-failover.enabled)
if [ "$(echo "$AUTOHA_ENABLED" | tr A-Z a-z)" = "true" ]; then
echo "Starting ZK Failover Controllers on NN hosts [$NAMENODES]"
"$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \
--config "$HADOOP_CONF_DIR" \
--hostnames "$NAMENODES" \
--script "$bin/hdfs" start zkfc
fi

# eof

我们看到上面红字部分，注释掉就不会起来了。


切忌在做这个操作前，自己先备份呢一份

非常感谢，曾经也想过这个方法，一直在找是不是有其他方面的参数可以来配置。

临云 发表于 2014-6-26 13:51
非常感谢，曾经也想过这个方法，一直在找是不是有其他方面的参数可以来配置。

SECONDARY_NAMENODES=$($HADOOP_PREFIX/bin/hdfs getconf -secondarynamenodes 2>/dev/null)
这是进入启动secondarynamenode的条件，你打印出来看看，就知道配置在什么地方了

这个版主太牛了！非常感谢！

版主写的非常详细谢谢啦~~~

请问如果是多个secondarynamenode，hdfs-site.xml应该怎么写？

hadoop 2.7.2环境