------------------------------软件版本--------------------------------------
主要从事网页设计、PC网站建设（电脑版网站建设）、wap网站建设（手机版网站建设）、成都响应式网站建设、程序开发、微网站、成都微信小程序等，凭借多年来在互联网的打拼，我们在互联网网站建设行业积累了丰富的成都网站设计、网站建设、外贸网站建设、网络营销经验，集策划、开发、设计、营销、管理等多方位专业化运作于一体，具备承接不同规模与类型的建设项目的能力。| RHEL6.8 | hadoop2.8.1 | apache-maven-3.3.9 |
| | | |
| findbugs-1.3.9 | protobuf-2.5.0.tar.gz | jdk-8u45 |
------------------------------软件版本---------------------------------------
大数据概述 可以用“5V + 1C”来概括：
| Variety | 多样化 |
| Volume | 海量 |
| Velocity | 快速 |
| Vitality | 灵活 |
| Value | 价值性 |
| Complexity | 复杂 |
1.Hadoop生态
宏观: Hadoop为主的生态圈 hadoop flume........
狭义: apache hadoop hadoop.apache.org
2.Hadoop(存储+计算+资源和作业调度)
hadoop1.x
HDFS 存储
MapReduce 计算+资源和作业调度
hadoop2.x 企业正在用
HDFS 存储
MapReduce 计算
YARN 资源和作业调度平台 计算组件都会on yarn
hadoop3.x ???
2.Maven部署
blog
2.1解压
[root@hadoop1 softwore]# pwd
/opt/softwore
[root@hadoop1 softwore]# ls
apache-maven-3.3.9-bin.zip hadoop-2.8.1-src.tar.gz jdk-8u45-linux-x64.gz
findbugs-1.3.9.zip hadoop-2.8.1.tar.gz protobuf-2.5.0.tar.gz
[root@hadoop1 softwore]# unzip apache-maven-3.3.9-bin.zip --配置环境变量 [root@hadoop1 softwore]# vi /etc/profile
export MAVEN_HOME=/opt/software/apache-maven-3.3.9
export MAVEN_OPTS="-Xms256m -Xmx512m"
export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH [root@hadoop1 softwore]# source /etc/profile
[root@hadoop1 apache-maven-3.3.9]# mvn --version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /opt/software/apache-maven-3.3.9
Java version: 1.8.0_45, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_45/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-358.el6.x86_64", arch: "amd64", family: "unix" |
Maven部署为二进制部署，解压配置环境变量
2.2配置mavne目录
2.3查看配置文件和 解压我们准备好的仓库文件
3.Hadoop编译
编译hadoop 通过Requirements我看可以知道需要安装jdk maven findbugs protocolBuffer cmake Zlib openssl-devel 等软件
3.1解压
3.2查看pom.xml
3.3查看BUILDING.txt
Requirements:编译软件环境要求
* Unix System
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
* Zlib devel (if compiling native code)
* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
3.4 JDK部署
[root@hadoop1 softwore]# tar -zxvf jdk-8u45-linux-x64.gz -C /usr/java[root@hadoop1 softwore]# ls -ld /usr/java/*
drwxr-xr-x 8 root root 4096 12月 15 2016 /usr/java/djdk1.7.0_79
drwxr-xr-x 8 uucp 143 4096 4月 11 2015 /usr/java/jdk1.8.0_45
drwxr-xr-x 8 uucp 143 4096 10月 7 2015 /usr/java/jdk1.8.0_65
[root@hadoop1 softwore]# vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_45
export JRE_HOME=/usr/java/jdk1.8.0_45/jre [root@hadoop1 softwore]# source /etc/profile
[root@hadoop1 softwore]# java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode) |
二进制版本解压，配置环境变量即可
3.5 MAVEN
上面我们已经安装maven和jdk检测版本是否满足我们编译软件要求
[root@hadoop1 softwore]# mvn --version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /opt/software/apache-maven-3.3.9
Java version: 1.8.0_45, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_45/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-431.el6.x86_64", arch: "amd64", family: "unix"
[root@hadoop1 softwore]#
3.6 FINDBUGS
[root@hadoop1 softwore]# unzip findbugs-1.3.9.zip
[root@hadoop1 softwore]# vi /etc/profile
export FINDBUGS_HOME=/opt/software/findbugs-1.3.9
export PATH=$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH [root@hadoop1 softwore]# source /etc/profile
[root@hadoop1 softwore]# findbugs -version
1.3.9 |
二进制安装解压，配置环境变量即可
3.7 PROTOCBUF编译安装
[root@hadoop1 software]# tar xvf protobuf-2.5.0.tar.gz
[root@hadoop1 yum.repos.d]# yum install -y gcc gcc-c++ make cmake ---编译环境
[root@hadoop1 yum.repos.d]#cd /opt/software/protobuf-2.5.0
[root@hadoop1 protobuf-2.5.0]# ./configure --prefix=/usr/local/protobuf
[root@hadoop1 protobuf-2.5.0]# make && make install
[root@hadoop1 protobuf-2.5.0]# vim /etc/profile
export PROTOC_HOME=/usr/local/protobuf
export PATH=$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
[root@hadoop1 software]# source /etc/profile
[root@hadoop1 software]# protoc --version
libprotoc 2.5.0
[root@hadoop000 local]# |
编译安装完成，配置环境变量
3.8 OTHER 依赖包
yum install -y openssl openssl-devel svn ncurses-devel zlib-devel libtool
yum install -y snappy snappy-devel bzip2 bzip2-devel \n
zo lzo-devel lzop autoconf automake |
其他依赖包在系统安装ios都包含，执行yum安装即可
---------------------------------------------------------------------------------------------------------------------------------------
至此，编译hadoop需要的编译环境全部准备好，下面开启编译之路
---------------------------------------------------------------------------------------------------------------------------------------
3.9 编译
[root@hadoop1 software]# tar xvf hadoop-2.8.1-src.tar.gz
[root@hadoop1 software]# cd hadoop-2.8.1-src
[root@hadoop1 hadoop-2.8.1-src]# pwd
/opt/software/hadoop-2.8.1-src
[root@hadoop1 hadoop-2.8.1-src]#
mvn clean package -Pdist,native -DskipTests –Dtar /opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz ----编译好软件包保存位置 |
直接编译需要联网找依赖包，一般网络情况较好，需要几个小时
提前准备好依赖包编译方式
[root@hadoop1 software]# tar xvf hadoop-2.8.1-src.tar.gz
[root@hadoop1 software]# cd hadoop-2.8.1-src
[root@hadoop1 hadoop-2.8.1-src]# pwd
/opt/software/hadoop-2.8.1-src [root@hadoop1 hadoop-2.8.1-src]# pwd
/opt/software/hadoop-2.8.1-src
[root@hadoop1 hadoop-2.8.1-src]# ls pom.xml ---根据这个文件下载jar
pom.xml
[root@hado -rw-r--r-- 1 root root 96721446 4月 25 09:57 .m2.tar.gz ---提前下好的仓库软件包 [root@hadoop1 software]#tar xvf .m2.tar.gz –C /root ---不想download，使用提前准备好的仓库软件--->解压----->root家目录 [root@hadoop1 hadoop-2.8.1-src]# mvn clean package -Pdist,native -DskipTests –Dtar …….. [INFO] Apache Hadoop Azure support ........................ SUCCESS [ 5.562 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 13.396 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 1.969 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 6.050 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 11.110 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 3.862 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.029 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:01 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13:45 min
[INFO] Finished at: 2018-04-25T14:03:20+08:00
[INFO] Final Memory: 161M/479M
[INFO] ------------------------------------------------------------------------ [root@hadoop1 hadoop-2.8.1-src]# 编译成功后的软件保存路径 [root@hadoop1 hadoop-2.8.1-src]# ls /opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-*
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1.jar
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1-javadoc.jar
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1-sources.jar
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1-test-sources.jar
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1:
bin etc include lib libexec LICENSE.txt NOTICE.txt README.txt sbin share
[root@hadoop1 hadoop-2.8.1-src]# du -sh /opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz
186M /opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz |
总结：完成源码的编译好打包，得到安装hadoop安装包，难点在于软件包的装备和仓库软件的准备，接下来准备开始hadoop安装之路。
另外有需要云服务器可以了解下创新互联scvps.cn，海内外云服务器15元起步，三天无理由+7*72小时售后在线，公司持有idc许可证，提供“云服务器、裸金属服务器、高防服务器、香港服务器、美国服务器、虚拟主机、免备案服务器”等云主机租用服务以及企业上云的综合解决方案，具有“安全稳定、简单易用、服务可用性高、性价比高”等特点与优势，专为企业上云打造定制，能够满足用户丰富、多元化的应用场景需求。
网站名称：大数据之----部署安装编译打包hadoop终极篇-创新互联
文章来源：
http://www.cdkjz.cn/article/dpocpe.html
