分享

hive on tez 指定tez.queue.name后执行报错



java.lang.IllegalStateException: There is conflicting local resource (htrace-core-3.2.0-incubating.jar) between dag local resource and vertex Reducer 2 local resource.
Resource of dag : resource { scheme: "hdfs" host: "ocetl" port: -1 file: "/tmp/hive/ocetl/4d0a684c-ee74-4f3d-b56d-7d4d126c318a/hive_2019-08-08_17-01-44_237_7985287463521422787-1/ocetl/_tez_scratch_dir/htrace-core-3.2.0-incubating.jar" } size: 1483913 timestamp: 1565254905777 type: FILE visibility: PRIVATE
Resource of vertex: resource { scheme: "hdfs" host: "ocetl" port: -1 file: "/tmp/hive/ocetl/_tez_session_dir/4d0a684c-ee74-4f3d-b56d-7d4d126c318a/htrace-core-3.2.0-incubating.jar" } size: 1483913 timestamp: 1565254866425 type: FILE visibility: PRIVATE
        at org.apache.tez.dag.api.DAG.verify(DAG.java:698)
        at org.apache.tez.dag.api.DAG.createDag(DAG.java:797)
        at org.apache.tez.client.TezClientUtils.prepareAndCreateDAGPlan(TezClientUtils.java:751)
        at org.apache.tez.client.TezClient.submitDAGSession(TezClient.java:571)
        at org.apache.tez.client.TezClient.submitDAG(TezClient.java:516)
        at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:379)
        at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:165)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2054)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1750)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1503)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1287)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1277)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:226)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:175)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:389)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:699)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:634)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:226)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:141)


已有(1)人评论

跳转到指定楼层
一曲离殇 发表于 2019-8-14 10:10:10
根据网上修改未成功TEZ-3703【https://community.hortonworks.co ... alangillegalst.html
自己修改【只校验大小,不校验时间戳】,目前已经可执行,是否存在其他问题待验证
[mw_shl_code=shell,true]vi tez-api/src/main/java/org/apache/tez/dag/api/DAG.java[/mw_shl_code]
[mw_shl_code=java,true]// 源代码
        if (commonTaskLocalFiles.containsKey(resourceName)
          && !commonTaskLocalFiles.get(resourceName).equals(resource)) {
          throw new IllegalStateException("There is conflicting local resource (" + resourceName
            + ") between dag local resource and vertex " + v.getName() + " local resource. "
            + "\nResource of dag : " + commonTaskLocalFiles.get(resourceName)
            + "\nResource of vertex: " + resource);
        }[/mw_shl_code]

[mw_shl_code=java,true]// 修改后的代码
        if (commonTaskLocalFiles.containsKey(resourceName)) {
          LocalResource commonResource = commonTaskLocalFiles.get(resourceName);
          if (!commonResource.equals(resource)) {
            String msg = "There is conflicting local resource (" + resourceName
              + ") between dag local resource and vertex " + v.getName() + " local resource. "
              + "\nResource of dag : " + commonResource + "\nResource of vertex: " + resource;
            if (resource.getSize() != commonResource.getSize()) {
              throw new IllegalStateException(msg);
            } else {
              LOG.error("[Default execution before same size. wangjw6] " + msg);
            }
          }
        }[/mw_shl_code]
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条