hive+impala+kudu+mysql：Communications link failure

今天出现一个诡异现象，在impala命令行中建表失败：[mw_shl_code=sql,true][apache.test.datanode1:21000] > create table test (id int primary key,name string) partition by hash partitions 8 stored as kudu;
Query: create table test (id int primary key,name string) partition by hash partitions 8 stored as kudu
ERROR: ImpalaRuntimeException: Error making 'createTable' RPC to Hive Metastore:
CAUSED BY: InvalidObjectException: kudu_db: Communications link failure

The last packet successfully received from the server was 1,404,095 milliseconds ago.  The last packet sent successfully to the server was 2 milliseconds ago.[/mw_shl_code]
报错说rpc通信失败，那么可以理解为元数据没有插入到mysql中

mysql显示结果：
[mw_shl_code=sql,true]mysql> select TBL_NAME,NAME from TBLS a inner join DBS b on a.DB_ID= b.DB_ID;
+----------------------------+-----------+
| TBL_NAME                | NAME    |
+----------------------------+-----------+
| kd_customer             | hive_db |
| kd_customer_10          | hive_db |
| kudu_test                | impala_db |
| kd_customer             | impala_db |
| kd_customer_parquet       | impala_db |
| kd_customer_text          | impala_db |
| kd_customer_withoutpart | kudu_db |
| kd_customer_withoutpart_10 | kudu_db |
| orders                   | kudu_db |
| test                      | kudu_db |
+----------------------------+-----------+
10 rows in set (0.00 sec)[/mw_shl_code]

hive命令行查询：
[mw_shl_code=sql,true]hive> show tables;
OK
test
Time taken: 0.057 seconds, Fetched: 1 row(s)[/mw_shl_code]

至此，证明mysql中是已经保存了这份元数据了，但是建表报错通信失败，那么再建一次：
[mw_shl_code=sql,true][apache.test.datanode1:21000] > create table kudu_db.test (id int primary key,name string) partition by hash partitions 8 stored as kudu;
Query: create table kudu_db.test (id int primary key,name string) partition by hash partitions 8 stored as kudu
ERROR: ImpalaRuntimeException: Error making 'createTable' RPC to Hive Metastore:
CAUSED BY: AlreadyExistsException: Table test already exists[/mw_shl_code]

说是test表已经存在，再来删除看看
[mw_shl_code=sql,true][apache.test.datanode1:21000] > drop table kudu_db.test;
Query: drop table kudu_db.test
ERROR: AnalysisException: Table does not exist: kudu_db.test[/mw_shl_code]
删除不成功，说是test表不在kudu_db中。那这奇怪了，两个报错冲突了，原因就在通信建立失败时引起的，
那mysql中已经有了元数据了为何又报错说通信失败呢？
先不跟踪源码，先解决下问题：
mysql中删除test表相关联的元数据：
先删除TABLE_PARAMS表中的数据，如果直接删除TBLS中数据，会报外键约束错误。
再大概介绍下kudu表的元数据在TABLE_PARAMS中长什么样子：
四个字段kudu.master_addresses、kudu.table_name、storage_handler、transient_lastDdlTime（都顾名思义的字段名不解释）
删除后再次在impala命令行中创建就ok了

总结：不算是疑难杂症，也很少会碰到，只是记录下