Hbase、elasticsearch整合中jar包沖突的問題解決

更新時(shí)間：2017年12月02日 14:22:35 作者：止魚

本篇文章主要介紹了Hbase、elasticsearch整合中jar包沖突的問題解決，小編覺得挺不錯(cuò)的，現(xiàn)在分享給大家，也給大家做個(gè)參考。一起跟隨小編過來看看吧

問題背景

再數(shù)據(jù)平臺(tái)中，項(xiàng)目搭建需要使用es和HBASE搭建數(shù)據(jù)查詢接口，整合的過程中出現(xiàn)jar包沖突的bug :com.google.common.base.Stopwatch.()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator

org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:239)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:150)
    ...
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
    at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntil
    ...
    at org.apache.hadoop.hbase.client.RegionServerCallable.prepare(RegionServerCallable.java:75)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:134)
    ... 45 more

解決辦法

經(jīng)排查，確認(rèn)是因?yàn)?com.google.guava 這個(gè)包引起的沖突。es依賴于18及以上版本，而HBASE只支持16及以上版本。而guava這個(gè)包從17開始內(nèi)部發(fā)生變化，方法發(fā)生改變，所以18并不會(huì)向下兼容16版本。在項(xiàng)目運(yùn)行的過程中，如果同時(shí)引入16、18版本，es及hbase的調(diào)用過程會(huì)發(fā)生混亂。那么接下來就好辦了，我們可以重新打包，將guava18打入es內(nèi)部，再在pom文件中顯示引用guava16版本的包。這樣，es調(diào)用包內(nèi)部打入的guava18，而hbase調(diào)用外部的guava16。

重新打包

新建maven工程，在pom文件中進(jìn)行如下配置：

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <!--
    this maven project is aim to fix the jar comflic in hbase & elasticsearch.
    in es, required guava 18+ or up. but in hbase you should use guava 16- or blow.

    cd the project directory and run 'sh ./cleanbuild.sh',you will get a new-self jar.
    then, adjust your project pom.xml whth:
    <dependency>
      <groupId>douguo.shaded.elasticsearch</groupId>
      <artifactId>douguo_shaded_elasticsearch</artifactId>
      <version>1.0-SNAPSHOT</version>
      <exclusions>
        <exclusion>
          <groupId>org.elasticsearch</groupId>
          <artifactId>elasticsearch</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    finally, your jar comflic will be fixed.

    @date:2017-11-30
    @author:zhangjianfei
    @since:douguo.shaded.elasticsearch-1.0.0
  -->
  <groupId>douguo.shaded.elasticsearch</groupId>
  <artifactId>douguo_shaded_elasticsearch</artifactId>
  <version>1.0-SNAPSHOT</version>

  <properties>
    <elasticsearch.version>2.4.1</elasticsearch.version>
  </properties>
  <dependencies>
    <dependency>
      <groupId>org.elasticsearch</groupId>
      <artifactId>elasticsearch</artifactId>
      <version>${elasticsearch.version}</version>
    </dependency>
    <dependency>
      <groupId>org.elasticsearch.plugin</groupId>
      <artifactId>shield</artifactId>
      <version>${elasticsearch.version}</version>
    </dependency>
  </dependencies>
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.4.1</version>
        <configuration>
          <createDependencyReducedPom>false</createDependencyReducedPom>
        </configuration>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <relocations>
                <relocation>
                  <pattern>com.google.guava</pattern>
                  <shadedPattern>douguo.shaded.elasticsearch.guava</shadedPattern>
                </relocation>
                <relocation>
                  <pattern>org.joda</pattern>
                  <shadedPattern>douguo.shaded.elasticsearch.joda</shadedPattern>
                </relocation>
                <relocation>
                  <pattern>com.google.common</pattern>
                  <shadedPattern>douguo.shaded.elasticsearch.common</shadedPattern>
                </relocation>
                <relocation>
                  <pattern>com.google.thirdparty</pattern>
                  <shadedPattern>douguo.shaded.elasticsearch.thirdparty</shadedPattern>
                </relocation>
              </relocations>
              <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer" />
              </transformers>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

  <repositories>
    <repository>
      <id>elasticsearch-releases</id>
      <url>http://maven.elasticsearch.org/releases</url>
      <releases>
        <enabled>true</enabled>
        <updatePolicy>daily</updatePolicy>
      </releases>
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
    </repository>
  </repositories>

</project>

將org.joda等4個(gè)可能有沖突的jar包通過maven-shade-plugin插件遷移后重新打個(gè)jar包從而使得在引入這個(gè)jar包時(shí)能夠使用該jar包自己的依賴而不是使用外部依賴。這里需要注意的是，需要將com.google.common等4個(gè)包全部重新遷移，否則會(huì)出現(xiàn)java.lang.IllegalAccessError: tried to access method com.google.common.base的錯(cuò)誤

項(xiàng)目打包

mvn clean install

新的依賴包會(huì)在.m2 maven倉(cāng)庫(kù)中，如果公司搭建了倉(cāng)庫(kù)的話，需要上傳jar包。如果直接運(yùn)行jar包的話，記得重新編譯項(xiàng)目，并替換lib目錄

項(xiàng)目載入新包

只需要在pom文件中配置：

<!-- douguo.shaded.elasticsearch -->
<dependency>
  <groupId>douguo.shaded.elasticsearch</groupId>
  <artifactId>douguo_shaded_elasticsearch</artifactId>
  <version>1.0-SNAPSHOT</version>
  <exclusions>
    <exclusion>
      <groupId>org.elasticsearch</groupId>
      <artifactId>elasticsearch</artifactId>
    </exclusion>
  </exclusions>
</dependency>


<!-- https://mvnrepository.com/artifact/com.google.guava/guava -->
<!-- this guava is only used in habse
   in es, 18.0+ is required, but hbase only supported 16.0 or blow.
   clean as install douguo.shaded.elasticsearch -->
<dependency>
  <groupId>com.google.guava</groupId>
  <artifactId>guava</artifactId>
  <version>16.0</version>
</dependency>

這樣，guava18包就在douguo.shaded.elasticsearch下，es會(huì)優(yōu)先調(diào)用。而外部配置的guava16就會(huì)被HBASE調(diào)用。2個(gè)版本的jar包互相獨(dú)立存在！

到此，問題就解決了！

以上就是本文的全部?jī)?nèi)容，希望對(duì)大家的學(xué)習(xí)有所幫助，也希望大家多多支持腳本之家。

您可能感興趣的文章: