文档平台

外观

Ambari-2.7.6 源码编译

下载源码 ​

下载 Ambari-2.7.6-sources.tar

shell
wget https://repo.huaweicloud.com/apache/ambari/ambari-2.7.6/apache-ambari-2.7.6-src.tar.gz

安装 python2.7,确保在终端执行 python 是 2.7 的环境（如环境安装python3 多个版本的一定要注意这条）。安装 python setuptools

shell
wget https://pypi.python.org/packages/45/29/8814bf414e7cd1031e1a3c8a4169218376e284ea2553cc0822a6ea1c2d78/setuptools-36.6.0.zip#md5=74663b15117d9a2cc5295d76011e6fd1

unzip setuptools-36.6.0.zip  # 附 unzip安装命令:yum install -y unzip zip
cd setuptools-36.6.0
python setup.py install

有部分依赖是 github，国内下载非常不稳定，可自行下载（迅雷、翻墙、或者其他第三方下载，下载后放到 Maven repo 对应目录下即可）

shell
wget https://github.com/yarnpkg/yarn/releases/download/v1.1.0/yarn-v1.1.0.tar.gz yarn-1.1.0.tar.gz
mv yarn-1.1.0.tar.gz $MAVEN_REPO/com/github/eirslett/yarn/1.1.0/

终于等到您！ hdp3.3.1(centos7_8)版本已完成，支持多个操作系统，支持x86_64和ARM64(aarch64)，欢迎下载试用，网盘链接：https://pan.baidu.com/s/1z_Yk-inzpZnOvtG8EHo_ow 提取码：wj68

准备工作 ​

  1. 安装依赖
  • 安装 jdk-1.8.291
  • 安装 maven 3.6.3
  • 安装 nodejs 14.21.3
  • 安装 文件服务器 jetty 或者 tomcat
  • 安装本地系统依赖库
shell
yum install -y rpm-build gcc-c++ python-devel git
yum install -y python-devel.x86_64
  1. 安装 npm/yarn 源

安装依赖库

shell
npm install -g bower
npm install -g yarn
npm install -g brunch@1.7.20
npm install -g bower
npm install -g gulp

设置淘宝源

shell
npm config set registry https://registry.npm.taobao.org
npm install -g cnpm --registry=https://registry.npm.taobao.org

yarn config set registry https://registry.npm.taobao.org -g
  1. 修改 ambari-admin/pom.xml 修改 bower 的路径地址
xml
<configuration>
  <workingDirectory>${basedir}/src/main/resources/ui/admin-web</workingDirectory>
  <executable>${basedir}/src/main/resources/ui/admin-web/node/${executable.node}</executable>
  <arguments>
    <!--
    <argument>${basedir}/src/main/resources/ui/admin-web/node_modules/bower/bin/bower</argument>
    -->
    <argument>bower</argument>
    <argument>install</argument>
    <argument>--allow-root</argument>
  </arguments>
</configuration>
  1. 修改 /ambari-metrics/pom.xml 内依赖的资源文件的下载路径

搭建内网文件服务器，使如下的版本的 hbase、hadoop等能下载到。

xml
<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <python.ver>python &gt;= 2.6</python.ver>
    <deb.python.ver>python (&gt;= 2.6)</deb.python.ver>
    <!--TODO change to HDP URL-->
    <!--
    <hbase.tar>https://archive.apache.org/dist/hbase/2.0.2/hbase-2.0.2-bin.tar.gz</hbase.tar>
    -->
    <hbase.tar>http://192.168.1.10/hdp/hbase-2.0.2-bin.tar.gz</hbase.tar>
    <hbase.folder>hbase-2.0.2</hbase.folder>

    <!--
    <hadoop.tar>https://archive.apache.org/dist/hadoop/common/hadoop-3.1.1/hadoop-3.1.1.tar.gz</hadoop.tar>
    -->
    <hadoop.tar>http://192.168.1.10/hdp/hadoop-3.1.1.tar.gz</hadoop.tar>
    <hadoop.folder>hadoop-3.1.1</hadoop.folder>
    <hadoop.version>3.1.1</hadoop.version>

    <!--
    <grafana.tar>https://dl.grafana.com/oss/release/grafana-6.7.4.linux-amd64.tar.gz</grafana.tar>
    -->
    <grafana.folder>grafana-6.7.4</grafana.folder>
    <grafana.tar>http://192.168.1.10/hdp/grafana-6.7.4.linux-amd64.tar.gz</grafana.tar>

    <!--
    <phoenix.tar>https://downloads.apache.org/phoenix/apache-phoenix-5.0.0-HBase-2.0/bin/apache-phoenix-5.0.0-HBase-2.0-bin.tar.gz</phoenix.tar>
    -->
    <phoenix.tar>http://192.168.1.10/hdp/apache-phoenix-5.0.0-HBase-2.0-bin.tar.gz</phoenix.tar>
    <phoenix.folder>apache-phoenix-5.0.0-HBase-2.0-bin</phoenix.folder>
    <!-- ... 其他配置 -->
<properties>
  1. 修改 ambari-logsearch/ambari-logsearch-web/pom.xml 依赖的 yarn 版本

主要是依赖的 yarn 版本太旧

xml
<properties>
    <logsearch.npm.config.tmp>/tmp/logsearch_npm_config_tmp</logsearch.npm.config.tmp>
    <node.version>v8.6.0</node.version>
    <!-- 将 1.1.0 修改为 1.20.0，版本太旧下载不到，且编译时问题比较多
    <yarn.version>v1.1.0</yarn.version>
    -->
    <yarn.version>v1.20.0</yarn.version>
</properties>
  1. 修改 ambari-infra-assembly/pom.xml 依赖的 solr 下载地址

提前下载好 solr-7.7.3，放置到文件服务器，使在内网能下载到。

xml
<properties>
    <mapping.base.path>/usr/lib</mapping.base.path>
    <!-- 修改 solr 下载地址
    <solr.tar>http://archive.apache.org/dist/lucene/solr/${solr.version}/solr-${solr.version}.tgz</solr.tar>
    -->
    <solr.tar>http://192.168.1.10/hdp/solr-${solr.version}.tgz</solr.tar>
</properties>
  1. 同步编辑过的文件
shell
scp ./ambari-metrics/pom.xml root@192.168.1.10:/home/hdp/build_work/apache-ambari-2.7.6-src/ambari-metrics/
scp ./ambari-admin/pom.xml root@192.168.1.10:/home/hdp/build_work/apache-ambari-2.7.6-src/ambari-admin/
scp ./ambari-metrics/ambari-metrics-storm-sink-legacy/pom.xml root@192.168.1.10:/home/hdp/build_work/apache-ambari-2.7.6-src/ambari-metrics/ambari-metrics-storm-sink-legacy/
scp ./ambari-metrics/ambari-metrics-timelineservice/pom.xml root@192.168.1.10:/home/hdp/build_work/apache-ambari-2.7.6-src/ambari-metrics/ambari-metrics-timelineservice/
scp ./ambari-logsearch/pom.xml root@192.168.1.10:/home/hdp/build_work/apache-ambari-2.7.6-src/ambari-logsearch/
scp ./ambari-logsearch/ambari-logsearch-web/pom.xml root@192.168.1.10:/home/hdp/build_work/apache-ambari-2.7.6-src/ambari-logsearch/ambari-logsearch-web
scp ./ambari-infra/pom.xml root@192.168.1.10:/home/hdp/build_work/apache-ambari-2.7.6-src/ambari-infra/
scp ./ambari-infra/ambari-infra-assembly/pom.xml root@192.168.1.10:/home/hdp/build_work/apache-ambari-2.7.6-src/ambari-infra/ambari-infra-assembly/
  1. 开始编译
shell
mvn -B clean install rpm:rpm -DnewVersion=2.7.6.0.0 -DbuildNumber=388e072381e71c7755673b7743531c03a4d61be8 -DskipTests -Drat.skip=true -Dpython.ver="python >= 2.6"
mvn -B install rpm:rpm -DnewVersion=2.7.6.0.0 -DbuildNumber=388e072381e71c7755673b7743531c03a4d61be8 -DskipTests -Drat.skip=true -Dpython.ver="python >= 2.6"
mvn -B install rpm:rpm -DnewVersion=2.7.6.0.0 -DbuildNumber=388e072381e71c7755673b7743531c03a4d61be8 -DskipTests -Drat.skip=true -Dpython.ver="python >= 2.6" -Dbuild-rpm -Dview

当所有模块编译成功，则 ambari 编译通过，如下。

编译成功

编译结果：

shell
mkdir /home/hdp/build_work/2.7.6
for f in `find . -size +5000000c |grep "noarch.rpm"`; do cp -fv $f /home/hdp/build_work/2.7.6; done
for f in `find . -size +5000000c |grep "x86_64.rpm"`; do cp -fv $f /home/hdp/build_work/2.7.6; done

问题解决 ​

  1. 无法访问org.json.simple.JSONAware

在ambari-metrics-storm-sink-legacy中pom.xml中添加依赖。

xml
<dependency>
      <groupId>com.googlecode.json-simple</groupId>
      <artifactId>json-simple</artifactId>
      <version>1.1</version>
</dependency>
  1. phantomjs 无法下载
log
[ERROR] error /root/apache-ambari-2.7.6-src/ambari-web/node_modules/phantomjs-prebuilt: Command failed.
[ERROR] Exit code: 1
[ERROR] Command: sh
[ERROR] Arguments: -c node install.js
[ERROR] Directory: /root/apache-ambari-2.7.6-src/ambari-web/node_modules/phantomjs-prebuilt
[ERROR] Output:
[ERROR] PhantomJS not found on PATH
[ERROR] Downloading https://github.com/Medium/phantomjs/releases/download/v2.1.1/phantomjs-2.1.1-linux-x86_64.tar.bz2
[ERROR] Saving to /tmp/npm_config_tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2
[ERROR] Receiving...
[ERROR]
[ERROR] Error making request.
[ERROR] Error: socket hang up
[ERROR]     at TLSSocket.onHangUp (_tls_wrap.js:1054:19)
[ERROR]     at TLSSocket.g (events.js:260:16)
[ERROR]     at emitNone (events.js:72:20)
[ERROR]     at TLSSocket.emit (events.js:166:7)
[ERROR]     at endReadableNT (_stream_readable.js:921:12)
[ERROR]     at nextTickCallbackWith2Args (node.js:442:9)
[ERROR]     at process._tickCallback (node.js:356:17)
[ERROR]
[ERROR] Please report this full log at https://github.com/Medium/phantomjs

手动下载，并放置到 tmp/npm_config_tmp/phantomjs/ 下

shell
cd /tmp/npm_config_tmp/phantomjs/
wget https://npm.taobao.org/mirrors/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2

# 如果采用其他下载工具下载后，可直接放到该目录
cp -v phantomjs-2.1.1-linux-x86_64.tar.bz2 /tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2
cp -v phantomjs-2.1.1-linux-x86_64.tar.bz2 /tmp/logsearch_npm_config_tmp/phantomjs/phantomjs-2.1.1-linux-x86_64.tar.bz2
  1. IClusterReporter 找不到

经过查证，在 maven repo 中 storm 0.10 系列根本没有 backtype.storm.metric.IClusterReporter 这个类。

shell
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on project ambari-metrics-storm-sink-legacy: Compilation failure: Compilation failure:
[ERROR] /home/hdp/build_work/apache-ambari-2.7.6-src/ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java:[37,29] 找不到符号
[ERROR]   符号:   类 IClusterReporter
[ERROR]   位置: 程序包 backtype.storm.metric
[ERROR] /home/hdp/build_work/apache-ambari-2.7.6-src/ambari-metrics/ambari-metrics-storm-sink-legacy/src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java:[42,16] 找不到符号
[ERROR]   符号: 类 IClusterReporter
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

经过分析，StormTimelineMetricsReporter 并无引用，且 storm 是 0.10 非常早期的版本，基本用不上，可删除。

shell
rm -vf ./src/main/java/org/apache/hadoop/metrics2/sink/storm/StormTimelineMetricsReporter.java

参考资料 ​