Hadoop伪分布式部署(HDFS)
来自 微信公众号 若泽大数据 qq交流群:707635769环境要求java、ssh[*]添加hadoop用户
# useradd hadoop# vi /etc/sudoers# 找到root ALL=(ALL) ALL，添加hadoop ALL=(ALL) NOPASSWD:ALL[*]上传并解压
# rz #上传hadoop-2.8.1.tar.gz# tar -xzvf hadoop-2.8.1.tar.gz[*]软连接
# ln -s /opt/software/hadoop-2.8.1 /opt/software/hadoop[*]设置环境变量
# vi /etc/profileexport HADOOP_HOME=/opt/software/hadoopexport PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH# source /etc/profile[*]设置用户、用户组
# chown -R hadoop:hadoop hadoop# chown -R hadoop:hadoop hadoop/*# chown -R hadoop:hadoop hadoop-2.8.1 # cd hadoop# rm -f *.txt[*]切换hadoop用户
# su - hadoop# lltotal 32drwxrwxr-x. 2 hadoop hadoop 4096 Jun2 14:24 bindrwxrwxr-x. 3 hadoop hadoop 4096 Jun2 14:24 etcdrwxrwxr-x. 2 hadoop hadoop 4096 Jun2 14:24 includedrwxrwxr-x. 3 hadoop hadoop 4096 Jun2 14:24 libdrwxrwxr-x. 2 hadoop hadoop 4096 Aug 20 13:59 libexecdrwxr-xr-x. 2 hadoop hadoop 4096 Aug 20 13:59 logsdrwxrwxr-x. 2 hadoop hadoop 4096 Jun2 14:24 sbindrwxrwxr-x. 4 hadoop hadoop 4096 Jun2 14:24 share # bin: 可执行文件# etc: 配置文件# sbin: shell脚本，启动关闭hdfs,yarn等[*]配置文件
# cd /opt/software/hadoop# vi etc/hadoop/core-site.xml<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://192.168.137.130:9000</value> # 配置自己机器的IP </property></configuration> # vi etc/hadoop/hdfs-site.xml<configuration> <property> <name>dfs.replication</name> <value>1</value> </property></configuration>[*]配置hadoop用户的ssh信任关系
# 公钥/密钥 配置无密码登录# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys# chmod 0600 ~/.ssh/authorized_keys # 查看日期，看是否配置成功# ssh hadoop-01 dateThe authenticity of host 'hadoop-01 (192.168.137.130)' can't be established.RSA key fingerprint is 09:f6:4a:f1:a0:bd:79:fd:34:e7:75:94:0b:3c:83:5a.Are you sure you want to continue connecting (yes/no)? yes # 第一次回车输入yesWarning: Permanently added 'hadoop-01,192.168.137.130' (RSA) to the list of known hosts.Sun Aug 20 14:22:28 CST 2017 # ssh hadoop-01 date #不需要回车输入yes,即OKSun Aug 20 14:22:29 CST 2017 # ssh localhost dateThe authenticity of host 'hadoop-01 (192.168.137.130)' can't be established.RSA key fingerprint is 09:f6:4a:f1:a0:bd:79:fd:34:e7:75:94:0b:3c:83:5a.Are you sure you want to continue connecting (yes/no)? yes # 第一次回车输入yesWarning: Permanently added 'hadoop-01,192.168.137.130' (RSA) to the list of known hosts.Sun Aug 20 14:22:28 CST 2017# ssh localhost date #不需要回车输入yes,即OKSun Aug 20 14:22:29 CST 2017[*]格式化和启动
# bin/hdfs namenode -format# sbin/start-dfs.shERROR: hadoop-01: Error: JAVA_HOME is not set and could not be found. localhost: Error: JAVA_HOME is not set and could not be found.解决方法:添加环境变量#vi etc/hadoop/hadoop-env.sh# 将export JAVA_HOME=${JAVA_HOME}改为export JAVA_HOME=/usr/java/jdk1.8.0_45 # sbin/start-dfs.shERROR: mkdir: cannot create directory `/opt/software/hadoop-2.8.1/logs': Permission denied解决方法:# exit# cd ../# chown -R hadoop:hadoop hadoop-2.8.1# su - hadoop# cd /opt/software/hadoop # 继续启动# sbin/start-dfs.sh[*]检查是否成功# jps
19536 DataNode19440 NameNode19876 Jps19740 SecondaryNameNode
访问： http://192.168.137.130:50070
[*]修改dfs启动的进程，以hadoop-01启动
启动的三个进程：namenode: hadoop-01 bin/hdfs getconf -namenodesdatanode: localhost datanodes (using default slaves file) etc/hadoop/slavessecondarynamenode: 0.0.0.0
# cd /opt/software/hadoop# echo"hadoop-01" > ./etc/hadoop/slaves # cat ./etc/hadoop/slaves hadoop-01 # vi ./etc/hadoop/hdfs-site.xml<property> <name>dfs.namenode.secondary.http-address</name> <value>hadoop-01:50090</value></property><property> <name>dfs.namenode.secondary.https-address</name> <value>hadoop-01:50091</value></property> # 重启# sbin/stop-dfs.sh# sbin/start-dfs.sh
页:
[1]