centos7 hadoop 集群 安装配置

张映 发表于 2018-12-27

分类目录: hadoop/spark/scala

标签:, , , ,

hadoop的集群先搞二台机器,一台管理机,一台node机,为什么呢。因为钱,机子也要钱。数据量是逐步增长起来的。如果一台node不能满足需求了,在增加node节点到集群。

在开始安装配置前,最好把该篇文章看上几遍,理顺了,在开始。特别是我踩过的坑。

一,服务器说明

10.0.0.237 bigserver1       //master
10.0.0.236 bigserver2      //datanode

二,修改主机名,并配置hosts

1,修改主机名

[root@localhost ~]# hostname
localhost.localdomain
[root@localhost ~]# hostname bigserver1
[root@localhost ~]# hostname
bigserver1

2, 在/etc/hosts文件中增加,所有节点一样

10.0.0.236 bigserver2
10.0.0.237 bigserver1

三,关闭防火墙和selinux

# systemctl stop firewalld      //停止
# systemctl disable firewalld   //取消启动

# cat /etc/sysconfig/selinux
SELINUX=disabled   //关闭

改完重启一下电脑。hadoop安装配置好了以后,防火墙可以打开,开放端口。

四,ssh免密码登录

# ssh-keygen -t rsa 

# ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.0.0.236 -p 22
# ssh-copy-id -i ~/.ssh/id_rsa.pub root@10.0.0.237 -p 22 

# scp ~/.ssh/id_rsa root@10.0.0.236:/root/.ssh/
# scp ~/.ssh/id_rsa root@10.0.0.237:/root/.ssh/

登录到236和237后
# cd ~/.ssh/
# chmod 600 id_rsa

我是在本机生成了公私钥,分别传到了236,237机器。

五,安装java1.8

# yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel

六,下载hadoop

https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.tar.gz

大家根据自己的需求去下载。

# tar zxvf hadoop-2.7.7.tar.gz
# mkdir /bigdata
# mv hadoop-2.2.7 /bigdata/hadoop

# mkdir -pv /bigdata/hadoop/{tmp,var,dfs}
# mkdir -pv /bigdata/hadoop/dfs/{name,data}

七,配置hadoop

1,备份

# cd /bigdata/hadoop/etc
# cp -r hadoop hadoop_bak

这一步很重要,养成一个良好的习惯会事半功倍。

2,配置core-site.xml

<property>
   <name>hadoop.tmp.dir</name>
   <value>/bigdata/hadoop/tmp</value>
</property>
<property>
   <name>fs.default.name</name>
   <value>hdfs://bigserver1:9000</value>
</property>

在<configuration></configuration>中添加

3,修改 hadoop-env.sh

# whereis javac
javac: /usr/bin/javac /usr/share/man/man1/javac.1.gz

# ll /usr/bin/javac
lrwxrwxrwx. 1 root root 23 Dec 27 00:08 /usr/bin/javac -> /etc/alternatives/javac

# ll /etc/alternatives/javac
lrwxrwxrwx. 1 root root 70 Dec 27 00:08 /etc/alternatives/javac -> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/bin/javac
//以上是查找环境变量

# echo "export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64" >> ~/.bashrc
# source ~/.bashrc
# vim hadoop-env.sh
将export JAVA_HOME=${JAVA_HOME}替换成
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64

如果不是管理工具包安装,填解压目录,即可

4,配置hdfs-site.xml

<property>
   <name>dfs.name.dir</name>
   <value>/bigdata/hadoop/dfs/name</value>
</property>
<property>
   <name>dfs.data.dir</name>
   <value>/bigdata/hadoop/dfs/data</value>
</property>
<property>
   <name>dfs.replication</name>
   <value>2</value>
</property>
<property>
   <name>dfs.permissions</name>
   <value>false</value>
</property>

5,配置mapred-site.xml

# cp mapred-site.xml.template mapred-site.xml
# vim mapred-site.xml
<property>
   <name>mapred.job.tracker</name>
   <value>bigserver1:49001</value>
</property>

<property>
   <name>mapred.local.dir</name>
   <value>/bigdata/hadoop/var</value>
</property>

<property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
</property>

6,修改slaves

# cat slaves
bigserver2

7,配置yarn-site.xml

<property>
   <name>yarn.resourcemanager.hostname</name>
   <value>bigserver1</value>
</property>
<property>
   <name>yarn.resourcemanager.address</name>
   <value>${yarn.resourcemanager.hostname}:8032</value>
</property>
<property>
   <name>yarn.resourcemanager.scheduler.address</name>
   <value>${yarn.resourcemanager.hostname}:8030</value>
</property>
<property>
   <name>yarn.resourcemanager.webapp.address</name>
   <value>${yarn.resourcemanager.hostname}:8088</value>
</property>
<property>
   <name>yarn.resourcemanager.webapp.https.address</name>
   <value>${yarn.resourcemanager.hostname}:8090</value>
</property>
<property>
   <name>yarn.resourcemanager.resource-tracker.address</name>
   <value>${yarn.resourcemanager.hostname}:8031</value>
</property>
<property>
   <name>yarn.resourcemanager.admin.address</name>
   <value>${yarn.resourcemanager.hostname}:8033</value>
</property>
<property>
   <name>yarn.nodemanager.aux-services</name>
   <value>mapreduce_shuffle</value>
</property>
<property>
   <name>yarn.nodemanager.vmem-check-enabled</name>
   <value>false</value>
</property>

不要轻易的去配置cpu,内存等。不然会影响mapredure。例如:

yarn.nodemanager.resource.memory-mb
yarn.scheduler.maximum-allocation-mb
yarn.nodemanager.vmem-pmem-ratio等

以下是我配置不全导致的错误 :

2018-12-27 09:11:54,178 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1545833322243_0001_m_000007_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-12-27 09:11:54,178 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1545833322243_0001_m_000008_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-12-27 09:11:54,178 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1545833322243_0001_r_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
2018-12-27 09:11:54,179 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:1024, vCores:1>
2018-12-27 09:11:54,185 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: reduceResourceRequest:<memory:1024, vCores:1>
2018-12-27 09:11:54,196 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1545833322243_0001, File: hdfs://bigserver1:9000/tmp/hadoop-yarn/staging/root/.staging/job_1545833322243_0001/job_1545833322243_0001_1.jhist
2018-12-27 09:11:55,138 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:1 ScheduledMaps:9 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
2018-12-27 09:11:55,194 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1545833322243_0001: ask=3 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:0, vCores:0> knownNMs=1
2018-12-27 09:11:55,195 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps
2018-12-27 09:11:56,198 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps
2018-12-27 09:11:57,202 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps

到这儿,hadoop就配置完成了,master节点和datanode节点配置一样。

八,初始化hadoop,并运行hadoop

1,只需要在master节点初始化,node节点不需要

# cd /bigdata/hadoop/bin/
./hadoop namenode -format

初始化成功后,会/bigdata/hadoop/dfs/name多出一个current文件夹。初始化一次后,最好不要在重新初始化,最好不要在重新初始化,最好不要在重新初始化。重要的事情说三遍。会导致master节点和datanode节点对不上。后面会具体说明。

2,只需要在master启动hadoop

# cd /bigdata/hadoop/sbin/
# ./start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [bigserver1]
bigserver1: starting namenode, logging to /home/bigdata/hadoop/logs/hadoop-root-namenode-bigserver1.out
bigserver2: starting datanode, logging to /home/bigdata/hadoop/logs/hadoop-root-datanode-bigserver2.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/bigdata/hadoop/logs/hadoop-root-secondarynamenode-bigserver1.out
starting yarn daemons
starting resourcemanager, logging to /home/bigdata/hadoop/logs/yarn-root-resourcemanager-bigserver1.out
bigserver2: starting nodemanager, logging to /home/bigdata/hadoop/logs/yarn-root-nodemanager-bigserver2.out

3,检查hadoop集群各节点是否正常启动

//master节点
[root@bigserver1 name]# netstat -tpnl |grep java
tcp        0      0 0.0.0.0:50070           0.0.0.0:*               LISTEN      5573/java
tcp        0      0 10.0.0.237:9000         0.0.0.0:*               LISTEN      5573/java
tcp        0      0 0.0.0.0:50090           0.0.0.0:*               LISTEN      5768/java
tcp6       0      0 10.0.0.237:8088         :::*                    LISTEN      5930/java
tcp6       0      0 10.0.0.237:8030         :::*                    LISTEN      5930/java
tcp6       0      0 10.0.0.237:8031         :::*                    LISTEN      5930/java
tcp6       0      0 10.0.0.237:8032         :::*                    LISTEN      5930/java
tcp6       0      0 10.0.0.237:8033         :::*                    LISTEN      5930/java
[root@bigserver1 name]# jps
3457 RunJar
6851 Jps
5573 NameNode
5768 SecondaryNameNode
5930 ResourceManager

//datanode节点
[root@bigserver2 sbin]# netstat -tpnl |grep java
tcp        0      0 127.0.0.1:44205         0.0.0.0:*               LISTEN      3405/java
tcp        0      0 0.0.0.0:50010           0.0.0.0:*               LISTEN      3405/java
tcp        0      0 0.0.0.0:50075           0.0.0.0:*               LISTEN      3405/java
tcp        0      0 0.0.0.0:50020           0.0.0.0:*               LISTEN      3405/java
tcp6       0      0 :::43959                :::*                    LISTEN      3520/java
tcp6       0      0 :::13562                :::*                    LISTEN      3520/java
tcp6       0      0 :::8040                 :::*                    LISTEN      3520/java
tcp6       0      0 :::8042                 :::*                    LISTEN      3520/java
[root@bigserver2 sbin]# jps
3520 NodeManager
5761 Jps
3405 DataNode

jps显示的内容,如果少了一个说明没有配置成功。如果master和datanode节点进程缺少也说明没有成功。

如果都没有什么问题的话,可以通过url来访问了。

http://10.0.0.237:50070,节点健康检查工具

http://10.0.0.237:8088,集群各节点任务分析工具

如下图

hadoop 健康检查

hadoop 健康检查

hadoop集群

hadoop集群

4,只需要在master停止hadoop

root@localhost sbin]# ./stop-all.sh   //停止
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
Stopping namenodes on [bigserver1]
bigserver1: stopping namenode
bigserver2: no datanode to stop   //刚开始配置时,datanode没有启动成功报的错
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
stopping yarn daemons
stopping resourcemanager
bigserver2: stopping nodemanager
no proxyserver to stop

通过jps查看,缺少了DataNode。但是nodemanager是起来的。问题出在master和datanode节点,集群点没有对上,这让我想起了mysql replication position,对不上也会出现无法同步的问题。导致no datanode to stop这个原因的产生,猜测是master节点,进行了多次的初始化。hadoop namenode -format。

解决办法如下:

master点,打开/bigdata/hadoop/dfs/name/current/VERSION,
datanode点,打开/bigdata/hadoop/dfs/data/current/VERSION,
将master节点的clusterID,copy到datanode中,重启就好。

网上查了一下,有人说同步namespaceID也可以,但是我用hadoop2.7.7版本中,datanode节点,/bigdata/hadoop/dfs/data/current/VERSION文件中根本没有namespaceID,我又不想加。哈哈。也不确定这样行不行。

九,测试hadoop集群

1,master节点hdfs创建测试目录

# ./bin/hdfs dfs -mkdir /test
# ./bin/hdfs dfs -ls /
Found 3 items
drwxr-xr-x - root supergroup 0 2018-12-26 20:42 /test
drwx------ - root supergroup 0 2018-12-26 21:27 /tmp
drwxr-xr-x - root supergroup 0 2018-12-26 20:20 /user

2,master节点上传测试文件到hdfs

# ./bin/hdfs dfs -put ./etc/hadoop/*.xml /test/

3,master节点测试mapredure

# ./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar grep /test/ ./output 'dfs[a-z.]+'
18/12/27 09:11:46 INFO client.RMProxy: Connecting to ResourceManager at bigserver1/10.0.0.237:8032
18/12/27 09:11:48 INFO input.FileInputFormat: Total input paths to process : 9
18/12/27 09:11:48 INFO mapreduce.JobSubmitter: number of splits:9
18/12/27 09:11:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1545833322243_0001
18/12/27 09:11:49 INFO impl.YarnClientImpl: Submitted application application_1545833322243_0001  //datanode端userlogs中有日志
18/12/27 09:11:49 INFO mapreduce.Job: The url to track the job: http://bigserver1:8088/proxy/application_1545833322243_0001/
18/12/27 09:11:49 INFO mapreduce.Job: Running job: job_1545833322243_0001
18/12/27 09:11:55 INFO mapreduce.Job: Job job_1545833322243_0001 running in uber mode : false
18/12/27 09:11:55 INFO mapreduce.Job: map 0% reduce 0%  //这块卡死,map reduce都是0

到datanode节点查看日志:

# cd /bigdata/hadoop/logs/userlogs

[root@bigserver2 userlogs]# ls
application_1545825824765_0001 application_1545827800765_0001 application_1545828806710_0001 application_1545829094007_0001 application_1545833322243_0001

[root@bigserver2 userlogs]# ll |grep application_1545833322243_0001
drwx--x--- 3 root root 52 12月 27 09:11 application_1545833322243_0001

[root@bigserver2 userlogs]# cd application_1545833322243_0001
[root@bigserver2 application_1545833322243_0001]# cd container_1545833322243_0001_01_000001/

[root@bigserver2 container_1545833322243_0001_01_000001]# ls
stderr stdout syslog

[root@bigserver2 container_1545833322243_0001_01_000001]# tail -f syslog
2018-12-27 09:14:12,652 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps
2018-12-27 09:14:13,654 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps
2018-12-27 09:14:14,657 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps
2018-12-27 09:14:15,659 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps
2018-12-27 09:14:16,662 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps
2018-12-27 09:14:17,665 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 1 due to lack of space for maps
。。。。。。。。。。。。。。。。。。。。忽略。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。

解决办法:上面也提到了,就是yarn-site.xml,配置内存和cpu相关的去掉。重启hadoop就好

看一下成功后

mapredure测试成功

mapredure测试成功

十,查日志

hadoop的日志,还是很多的,还没有装hbase,hive,spark等。除了进入服务器查看外,还可以通过网页查看。

datanode logs 页面访问

datanode logs 页面访问

master log页面

master log页面

不怕问题,就怕出了问题,不知道错在哪里。随便点点网页log就发现个问题

2018-12-27 10:23:04,569 ERROR org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in doCheckpoint
java.io.IOException: Inconsistent checkpoint fields.
LV = -63 namespaceID = 1131284630 cTime = 0 ; clusterId = CID-ea915c79-c5cb-4d23-bc55-e4530f999cb0 ; blockpoolId = BP-508509447-10.0.0.237-1545809802003.
Expecting respectively: -63; 839710719; 0; CID-66e894a8-1cb1-4b8e-bac4-2bc3b526e062; BP-262790598-10.0.0.237-1545803654780.
at org.apache.hadoop.hdfs.server.namenode.CheckpointSignature.validateStorageInfo(CheckpointSignature.java:134)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:531)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:395)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$1.run(SecondaryNameNode.java:361)
at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:415)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:357)
at java.lang.Thread.run(Thread.java:748)

解决办法:

mater节点,删除该目录/bigdata/hadoop/tmp/dfs/namesecondary/current下的所有文件,重启hadoop即可。



转载请注明
作者:海底苍鹰
地址:http://blog.51yip.com/hadoop/2013.html

1 条评论

  1. laihuibiao 留言

    启动的时候,报错:ssh: connect to host master port 22: Connection timed out