|
编译类的帖子都不通用，还是希望能给大家提供些帮助：
MAC OS环境下编译，首先要看一手资料：
下载源码包，解压，进入目录，vi BUILDING.txt
[backcolor=rgba(0, 0, 0, 0.9)] 1 Build instructions for Hadoop [backcolor=rgba(0, 0, 0, 0.9)] 2 [backcolor=rgba(0, 0, 0, 0.9)] 3 ---------------------------------------------------------------------------- ------ [backcolor=rgba(0, 0, 0, 0.9)] 4 Requirements: [backcolor=rgba(0, 0, 0, 0.9)] 5 [backcolor=rgba(0, 0, 0, 0.9)] 6 * Unix System [backcolor=rgba(0, 0, 0, 0.9)] 7 * JDK 1.8 [backcolor=rgba(0, 0, 0, 0.9)] 8 * Maven 3.3 or later [backcolor=rgba(0, 0, 0, 0.9)] 9 * Protocol Buffers 3.7.1 (if compiling native code) [backcolor=rgba(0, 0, 0, 0.9)] 10 * CMake 3.1 or newer (if compiling native code) [backcolor=rgba(0, 0, 0, 0.9)] 11 * Zlib devel (if compiling native code) [backcolor=rgba(0, 0, 0, 0.9)] 12 * Cyrus SASL devel (if compiling native code) [backcolor=rgba(0, 0, 0, 0.9)] 13 * One of the compilers that support thread_local storage: GCC 4.8.1 or later , Visual Studio, [backcolor=rgba(0, 0, 0, 0.9)] 14 Clang (community version), Clang (version for iOS 9 and later) (if compili ng native code) [backcolor=rgba(0, 0, 0, 0.9)] 15 * openssl devel (if compiling native hadoop-pipes and to get the best HDFS e ncryption performance) [backcolor=rgba(0, 0, 0, 0.9)] 16 * Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fu se_dfs) [backcolor=rgba(0, 0, 0, 0.9)] 17 * Doxygen ( if compiling libhdfspp and generating the documents ) [backcolor=rgba(0, 0, 0, 0.9)] 18 * Internet connection for first build (to fetch all Maven and Hadoop depende ncies) [backcolor=rgba(0, 0, 0, 0.9)] 19 * python (for releasedocs) [backcolor=rgba(0, 0, 0, 0.9)] 20 * bats (for shell code testing) [backcolor=rgba(0, 0, 0, 0.9)] 21 * Node.js / bower / Ember-cli (for YARN UI v2 building)
讲的很清楚，一步一步操作，OS,maven,jdk,cmake,zlib,cyrus-sasl,gcc,openssl这些没什么好说的，照着装好配好就行，注意文档中说的protocol版本是3.7.1
[backcolor=rgba(0, 0, 0, 0.9)] 23 ---------------------------------------------------------------------------------- [backcolor=rgba(0, 0, 0, 0.9)] 24 The easiest way to get an environment with all the appropriate tools is by means [backcolor=rgba(0, 0, 0, 0.9)] 25 of the provided Docker config. [backcolor=rgba(0, 0, 0, 0.9)] 26 This requires a recent version of docker (1.4.1 and higher are known to work). [backcolor=rgba(0, 0, 0, 0.9)] 27 [backcolor=rgba(0, 0, 0, 0.9)] 28 On Linux / Mac: [backcolor=rgba(0, 0, 0, 0.9)] 29 Install Docker and run this command: [backcolor=rgba(0, 0, 0, 0.9)] 30 [backcolor=rgba(0, 0, 0, 0.9)] 31 $ ./start-build-env.sh [backcolor=rgba(0, 0, 0, 0.9)] 32 [backcolor=rgba(0, 0, 0, 0.9)] 33 The prompt which is then presented is located at a mounted version of the source tree [backcolor=rgba(0, 0, 0, 0.9)] 34 and all required tools for testing and building have been installed and configured. [backcolor=rgba(0, 0, 0, 0.9)] 35 [backcolor=rgba(0, 0, 0, 0.9)] 36 Note that from within this docker environment you ONLY have access to the Hadoop source [backcolor=rgba(0, 0, 0, 0.9)] 37 tree from where you started. So if you need to run [backcolor=rgba(0, 0, 0, 0.9)] 38 dev-support/bin/test-patch /path/to/my.patch [backcolor=rgba(0, 0, 0, 0.9)] 39 then the patch must be placed inside the hadoop source tree. [backcolor=rgba(0, 0, 0, 0.9)] 40 [backcolor=rgba(0, 0, 0, 0.9)] 41 Known issues: [backcolor=rgba(0, 0, 0, 0.9)] 42 - On Mac with Boot2Docker the performance on the mounted directory is currently extremely slow. [backcolor=rgba(0, 0, 0, 0.9)] 43 This is a known problem related to boot2docker on the Mac. [backcolor=rgba(0, 0, 0, 0.9)] 44 See: [backcolor=rgba(0, 0, 0, 0.9)] 46 This issue has been resolved as a duplicate, and they point to a new feature for utilizing NFS mounts [backcolor=rgba(0, 0, 0, 0.9)] 47 as the proposed solution: [backcolor=rgba(0, 0, 0, 0.9)] 49 An alternative solution to this problem is to install Linux native inside a virtual machine [backcolor=rgba(0, 0, 0, 0.9)] 50 and run your IDE and Docker etc inside that VM. [backcolor=rgba(0, 0, 0, 0.9)] 51 [backcolor=rgba(0, 0, 0, 0.9)] 52 ----------------------------------------------------------------------------------
这段说的很清楚了，先把docker环境装了，启动docker，接下来一条命令：./start-build-env.sh,剩下的交给docker吧
假如你想通过传统的方式，比如：
[backcolor=rgba(0, 0, 0, 0.9)]326 ---------------------------------------------------------------------------------- [backcolor=rgba(0, 0, 0, 0.9)]327 Building distributions: [backcolor=rgba(0, 0, 0, 0.9)]328 [backcolor=rgba(0, 0, 0, 0.9)]329 Create binary distribution without native code and without documentation: [backcolor=rgba(0, 0, 0, 0.9)]330 [backcolor=rgba(0, 0, 0, 0.9)]331 $ mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true [backcolor=rgba(0, 0, 0, 0.9)]332 [backcolor=rgba(0, 0, 0, 0.9)]333 Create binary distribution with native code and with documentation: [backcolor=rgba(0, 0, 0, 0.9)]334 [backcolor=rgba(0, 0, 0, 0.9)]335 $ mvn package -Pdist,native,docs -DskipTests -Dtar [backcolor=rgba(0, 0, 0, 0.9)]336 [backcolor=rgba(0, 0, 0, 0.9)]337 Create source distribution: [backcolor=rgba(0, 0, 0, 0.9)]338 [backcolor=rgba(0, 0, 0, 0.9)]339 $ mvn package -Psrc -DskipTests [backcolor=rgba(0, 0, 0, 0.9)]340 [backcolor=rgba(0, 0, 0, 0.9)]341 Create source and binary distributions with native code and documentation: [backcolor=rgba(0, 0, 0, 0.9)]342 [backcolor=rgba(0, 0, 0, 0.9)]343 $ mvn package -Pdist,native,docs,src -DskipTests -Dtar [backcolor=rgba(0, 0, 0, 0.9)]344 [backcolor=rgba(0, 0, 0, 0.9)]345 Create a local staging version of the website (in /tmp/hadoop-site) [backcolor=rgba(0, 0, 0, 0.9)]346 [backcolor=rgba(0, 0, 0, 0.9)]347 $ mvn clean site -Preleasedocs; mvn site:stage -DstagingDirectory=/tmp/hadoop-site [backcolor=rgba(0, 0, 0, 0.9)]348 [backcolor=rgba(0, 0, 0, 0.9)]349 Note that the site needs to be built in a second pass after other artifacts. [backcolor=rgba(0, 0, 0, 0.9)]350 [backcolor=rgba(0, 0, 0, 0.9)]351 ----------------------------------------------------------------------------------
或者：
[backcolor=rgba(0, 0, 0, 0.9)]381 ---------------------------------------------------------------------------------- [backcolor=rgba(0, 0, 0, 0.9)]382 Installing required dependencies for clean install of macOS 10.14: [backcolor=rgba(0, 0, 0, 0.9)]383 [backcolor=rgba(0, 0, 0, 0.9)]384 * Install Xcode Command Line Tools [backcolor=rgba(0, 0, 0, 0.9)]385 $ xcode-select --install [backcolor=rgba(0, 0, 0, 0.9)]386 * Install Homebrew [backcolor=rgba(0, 0, 0, 0.9)]388 * Install OpenJDK 8 [backcolor=rgba(0, 0, 0, 0.9)]389 $ brew tap AdoptOpenJDK/openjdk [backcolor=rgba(0, 0, 0, 0.9)]390 $ brew cask install adoptopenjdk8 [backcolor=rgba(0, 0, 0, 0.9)]391 * Install maven and tools [backcolor=rgba(0, 0, 0, 0.9)]392 $ brew install maven autoconf automake cmake wget [backcolor=rgba(0, 0, 0, 0.9)]393 * Install native libraries, only openssl is required to compile native code, [backcolor=rgba(0, 0, 0, 0.9)]394 you may optionally install zlib, lz4, etc. [backcolor=rgba(0, 0, 0, 0.9)]395 $ brew install openssl [backcolor=rgba(0, 0, 0, 0.9)]396 * Protocol Buffers 3.7.1 (required to compile native code) [backcolor=rgba(0, 0, 0, 0.9)]398 $ mkdir -p protobuf-3.7 && tar zxvf protobuf-java-3.7.1.tar.gz --strip-components 1 -C protobuf-3.7 [backcolor=rgba(0, 0, 0, 0.9)]399 $ cd protobuf-3.7 [backcolor=rgba(0, 0, 0, 0.9)]400 $ ./configure [backcolor=rgba(0, 0, 0, 0.9)]401 $ make [backcolor=rgba(0, 0, 0, 0.9)]402 $ make check [backcolor=rgba(0, 0, 0, 0.9)]403 $ make install [backcolor=rgba(0, 0, 0, 0.9)]404 $ protoc --version [backcolor=rgba(0, 0, 0, 0.9)]405 [backcolor=rgba(0, 0, 0, 0.9)]406 Note that building Hadoop 3.1.1/3.1.2/3.2.0 native code from source is broken [backcolor=rgba(0, 0, 0, 0.9)]407 on macOS. For 3.1.1/3.1.2, you need to manually backport YARN-8622. For 3.2.0, [backcolor=rgba(0, 0, 0, 0.9)]408 you need to backport both YARN-8622 and YARN-9487 in order to build native code. [backcolor=rgba(0, 0, 0, 0.9)]409 [backcolor=rgba(0, 0, 0, 0.9)]410 ----------------------------------------------------------------------------------
下载的maven依赖确实有点多，网络也要好才行，作者是搞了个vpn编译的
[backcolor=rgba(0, 0, 0, 0.9)]zhuqitian@zhuqitiandeMacBook-Pro hadoop-3.3.0-src % du -sh ~/.m2 [backcolor=rgba(0, 0, 0, 0.9)]893M /Users/zhuqitian/.m2
最终的结果是这样的：
[backcolor=rgba(0, 0, 0, 0.9)][INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 0.550 s] [backcolor=rgba(0, 0, 0, 0.9)][INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.025 s] [backcolor=rgba(0, 0, 0, 0.9)][INFO] Apache Hadoop Client API ........................... SUCCESS [01:08 min] [backcolor=rgba(0, 0, 0, 0.9)][INFO] Apache Hadoop Client Runtime ....................... SUCCESS [01:02 min] [backcolor=rgba(0, 0, 0, 0.9)][INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [ 0.673 s] [backcolor=rgba(0, 0, 0, 0.9)][INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [01:59 min] [backcolor=rgba(0, 0, 0, 0.9)][INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [ 0.415 s] [backcolor=rgba(0, 0, 0, 0.9)][INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [ 0.107 s] [backcolor=rgba(0, 0, 0, 0.9)][INFO] Apache Hadoop Distribution ......................... SUCCESS [ 0.228 s] [backcolor=rgba(0, 0, 0, 0.9)][INFO] Apache Hadoop Client Modules ....................... SUCCESS [ 0.023 s] [backcolor=rgba(0, 0, 0, 0.9)][INFO] Apache Hadoop Tencent COS Support .................. SUCCESS [ 0.416 s] [backcolor=rgba(0, 0, 0, 0.9)][INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 0.260 s] [backcolor=rgba(0, 0, 0, 0.9)][INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.023 s] [backcolor=rgba(0, 0, 0, 0.9)][INFO] ------------------------------------------------------------------------ [backcolor=rgba(0, 0, 0, 0.9)][INFO] BUILD SUCCESS [backcolor=rgba(0, 0, 0, 0.9)][INFO] ------------------------------------------------------------------------ [backcolor=rgba(0, 0, 0, 0.9)][INFO] Total time: 06:47 min [backcolor=rgba(0, 0, 0, 0.9)][INFO] Finished at: 2020-08-25T18:36:38+08:00 [backcolor=rgba(0, 0, 0, 0.9)][INFO] ------------------------------------------------------------------------
然后hadoop-dist/target下会生成个gz文件：
[backcolor=rgba(0, 0, 0, 0.9)]zhuqitian@zhuqitiandeMacBook-Pro hadoop-3.3.0-src % ls -lh hadoop-dist/target [backcolor=rgba(0, 0, 0, 0.9)]total 885376 [backcolor=rgba(0, 0, 0, 0.9)]drwxr-xr-x 3 zhuqitian staff 96B 8 25 18:36 antrun [backcolor=rgba(0, 0, 0, 0.9)]drwxr-xr-x 3 zhuqitian staff 96B 8 25 18:36 classes [backcolor=rgba(0, 0, 0, 0.9)]drwxr-xr-x 14 zhuqitian staff 448B 8 25 20:52 hadoop-3.3.0 [backcolor=rgba(0, 0, 0, 0.9)]-rw-r--r-- 1 zhuqitian staff 430M 8 25 20:52 hadoop-3.3.0.tar.gz [backcolor=rgba(0, 0, 0, 0.9)]drwxr-xr-x 3 zhuqitian staff 96B 8 25 20:47 hadoop-tools-deps [backcolor=rgba(0, 0, 0, 0.9)]drwxr-xr-x 3 zhuqitian staff 96B 8 25 18:36 maven-shared-archive-resources [backcolor=rgba(0, 0, 0, 0.9)]drwxr-xr-x 3 zhuqitian staff 96B 8 25 18:36 test-classes [backcolor=rgba(0, 0, 0, 0.9)]drwxr-xr-x 2 zhuqitian staff 64B 8 25 18:36 test-dir
然后你就可以很方便的打开idea看源码了：
|
/2 
京ICP备2020039040号
简书网举报电话：021-34700000
提升卡
置顶卡
沉默卡
喧嚣卡
变色卡
千斤顶
显身卡
