問題背景
解決完FileNotExsit的問題后，重新Build Cube，發(fā)現(xiàn)在fact table distinct階段還是報錯，錯誤如下：
錯誤
java.lang.NoSuchMethodError: org.apache.hadoop.yarn.proto.YarnProtos$LocalResourceProto.hashLong(J)I
at org.apache.hadoop.yarn.proto.YarnProtos$LocalResourceProto.hashCode(YarnProtos.java:11864)
at org.apache.hadoop.yarn.api.records.impl.pb.LocalResourcePBImpl.hashCode(LocalResourcePBImpl.java:62)
at java.util.HashMap.hash(HashMap.java:338)
at java.util.HashMap.put(HashMap.java:611)
at org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:133)
at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:163)
at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at org.apache.kylin.engine.mr.common.AbstractHadoopJob.waitForCompletion(AbstractHadoopJob.java:149)
at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:108)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:92)
at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:120)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
問題定位
從異常來看，應(yīng)該是類的方法找不到，查看源碼，找到LocalResourceProto，發(fā)現(xiàn)方法hashLong(long):int是來自其子類AbstractMessage中
/**
* Helper method for implementing {@link Message#hashCode()}.
* @see Boolean#hashCode()
*/
protected static int hashLong(long n) {
return (int) (n ^ (n >>> 32));
}
奇怪了，明顯是存在，為什么還會出下NoSuchMethodError??想了一下，可能原因是版本沖突，修改kylin的啟動參數(shù)，在啟動參數(shù)中新增-versbose:class，查看類是從哪個jar包中加載的。修改${KYLIN_HOME}/bin下面的setenv.sh腳本。
export KYLIN_EXTRA_START_OPTS="-verbose:class"
重啟程序后查看類加載日志，截取的日志如下
kylin.out:[Loaded org.apache.logging.log4j.message.AbstractMessageFactory from file:/usr/local/apache-hive-2.1.1-bin/lib/log4j-api-2.4.1.jar]
kylin.out:[Loaded com.google.protobuf.AbstractMessageLite from file:/usr/local/apache-kylin-1.6.0-hbase1.x-bin/lib/kylin-jdbc-1.6.0.jar]
kylin.out:[Loaded com.google.protobuf.AbstractMessage from file:/usr/local/apache-kylin-1.6.0-hbase1.x-bin/lib/kylin-jdbc-1.6.0.jar]
kylin.out:[Loaded com.google.protobuf.AbstractMessageLite$Builder$LimitedInputStream from file:/usr/local/apache-kylin-1.6.0-hbase1.x-bin/lib/kylin-jdbc-1.6.0.jar]
kylin.out:[Loaded com.google.protobuf.AbstractMessageLite$Builder from file:/usr/local/apache-kylin-1.6.0-hbase1.x-bin/lib/kylin-jdbc-1.6.0.jar]
kylin.out:[Loaded com.google.protobuf.AbstractMessage$Builder from file:/usr/local/apache-kylin-1.6.0-hbase1.x-bin/lib/kylin-jdbc-1.6.0.jar]
發(fā)現(xiàn)類AbstractMessage是從包/usr/local/apache-kylin-1.6.0-hbase1.x-bin/lib/kylin-jdbc-1.6.0.jar加載的，而不是加載的hadoop classpath下的protobuf-java-2.5.0，由于查看了下kylin-jdbc工程的pom依賴
[INFO] --- maven-dependency-plugin:2.10:tree (default-cli) @ kylin-jdbc ---
[INFO] org.apache.kylin:kylin-jdbc:jar:1.6.0
[INFO] +- org.apache.calcite.avatica:avatica:jar:1.8.0:compile
[INFO] | +- org.apache.calcite.avatica:avatica-metrics:jar:1.8.0:compile
[INFO] | +- com.fasterxml.jackson.core:jackson-core:jar:2.6.3:compile
[INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.6.3:compile
[INFO] | +- com.fasterxml.jackson.core:jackson-databind:jar:2.2.4:compile
[INFO] | +- com.google.protobuf:protobuf-java:jar:3.0.0-beta-1:compile
[INFO] | \- org.apache.httpcomponents:httpcore:jar:4.4.4:compile
[INFO] +- org.apache.httpcomponents:httpclient:jar:4.2.5:compile
[INFO] | +- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] | \- commons-codec:commons-codec:jar:1.6:compile
[INFO] +- junit:junit:jar:4.12:test
[INFO] | \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] +- log4j:log4j:jar:1.2.17:provided
[INFO] +- org.slf4j:slf4j-log4j12:jar:1.7.21:provided
[INFO] +- org.slf4j:jcl-over-slf4j:jar:1.7.21:compile
[INFO] \- org.slf4j:slf4j-api:jar:1.7.21:compile
其依賴的是protobuf的3.0.0-beta-1版本，和hadoop中proto編譯時的版本沖突，于是一種解決方式時先暫時刪除kylin-jdbc-1.6.0.jar中protobuf的相關(guān)包
jar xvf kylin-jdbc-1.6.0.jar
rm -rf com/google/protobuf/*
rm -rf kylin-jdbc-1.6.0.jar
jar cvf kylin-jdbc-1.6.0.jar *
遺留問題
kylin依賴了Apache Calcite包，而它依賴protobuf-3.0.0-beta-1，如果按照上述的方式修改，如果calcite依賴了3.0.0的新特性，則也會出現(xiàn)相關(guān)的錯誤；一種解決方式是找一個兼容雙方的版本；但這種方式很難，比如Hadoop版本可能是公司級別，很難統(tǒng)一處理；另外一種方式是通過maven-shade-plugin來進(jìn)行相關(guān)處理，為不同版本的包加一個前綴。
