最近在玩flink，想让flink sql借用hive的元数据库。可是呢，flink sql能连hive的版本，到目前为止只有2个，hive 1.2.1和hive 2.3.4。
cdh6.3.1装的hive是2.1.1，所以不升级hive，flink sql根本用不了。
1，备份hive 2.1.1元数据库
# mysqldump -uroot -ptest metastore > ./metastore.sql
2，查看cdh6.3.1默认的hive版本
[root@bigserver1 lib]# hive --version WARNING: Use "yarn jar" to launch YARN applications. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Hive 2.1.1-cdh6.3.1 Subversion file:///container.redhat7/build/cdh/hive/2.1.1-cdh6.3.1/rpm/BUILD/hive-2.1.1-cdh6.3.1 -r e8d55f408b4f9aa2648bc9e34a8f802d53d6aab3 Compiled by jenkins on Thu Sep 26 03:03:11 PDT 2019 From source with checksum 75df142c785df1d0b37a9d40fe16169f
3，下载hive 2.3.4
# wget http://mirror.bit.edu.cn/apache/hive/hive-2.3.4/apache-hive-2.3.4-bin.tar.gz # tar zxvf apache-hive-2.3.4-bin.tar.gz -C /opt/cloudera/ # mv -r /opt/cloudera/apache-hive-2.3.4-bin /opt/cloudera/hive
注意：进行升级前，在clouderat管理后台，停止掉，根hive相关的应用。
4，复制hive2.3.4包到原来的hive中
# cp -r /opt/cloudera/hive/lib /opt/cloudera/parcels/CDH/lib/hive/lib234
5，修改启动hive文件，调用hive2.3.4 jar包
# vim /opt/cloudera/parcels/CDH/lib/hive/bin/hive
# HIVE_LIB=${HIVE_HOME}/lib //注释本行，改为
HIVE_LIB=${HIVE_HOME}/lib234
6，同步文件
# cd /opt/cloudera/parcels/CDH/lib/hive # scp -r lib234 bigserver4:/opt/cloudera/parcels/CDH/lib/hive # scp -r lib234 bigserver3:/opt/cloudera/parcels/CDH/lib/hive # scp -r lib234 bigserver2:/opt/cloudera/parcels/CDH/lib/hive # scp -r lib234 bigserver5:/opt/cloudera/parcels/CDH/lib/hive # scp bin/hive bigserver4:/opt/cloudera/parcels/CDH/lib/hive/bin # scp bin/hive bigserver3:/opt/cloudera/parcels/CDH/lib/hive/bin # scp bin/hive bigserver2:/opt/cloudera/parcels/CDH/lib/hive/bin # scp bin/hive bigserver5:/opt/cloudera/parcels/CDH/lib/hive/bin //所有节点创建软连接，并分配权限 # ln -s /opt/cloudera/parcels/CDH/lib/hive/lib234/jline-2.12.jar /opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib/jline-2.12.jar # chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcels
7，升级hive 2.1.1 元数据库 metastore
mysql> use metastore; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> source /opt/cloudera/hive/scripts/metastore/upgrade/mysql/upgrade-2.1.0-to-2.2.0.mysql.sql; +------------------------------------------------+ | | +------------------------------------------------+ | Upgrading MetaStore schema from 2.1.0 to 2.2.0 | +------------------------------------------------+ 1 row in set, 1 warning (0.00 sec) Query OK, 0 rows affected (0.05 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 30 rows affected (0.01 sec) Rows matched: 30 Changed: 30 Warnings: 0 Query OK, 0 rows affected (0.06 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.02 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.04 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.00 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 191 rows affected (0.04 sec) Records: 191 Duplicates: 0 Warnings: 0 Query OK, 77 rows affected (0.04 sec) Records: 77 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.03 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.01 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.03 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.01 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.00 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.00 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.03 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.01 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.00 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.00 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.01 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.00 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 0 rows affected (0.00 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 1 row affected (0.00 sec) Rows matched: 1 Changed: 1 Warnings: 0 +---------------------------------------------------------+ | | +---------------------------------------------------------+ | Finished upgrading MetaStore schema from 2.1.0 to 2.2.0 | +---------------------------------------------------------+ 1 row in set, 1 warning (0.00 sec) mysql> source /opt/cloudera/hive/scripts/metastore/upgrade/mysql/upgrade-2.2.0-to-2.3.0.mysql.sql; +------------------------------------------------+ | | +------------------------------------------------+ | Upgrading MetaStore schema from 2.2.0 to 2.3.0 | +------------------------------------------------+ 1 row in set, 1 warning (0.00 sec) Query OK, 0 rows affected (0.01 sec) Records: 0 Duplicates: 0 Warnings: 0 Query OK, 1 row affected (0.01 sec) Rows matched: 1 Changed: 1 Warnings: 0 +---------------------------------------------------------+ | | +---------------------------------------------------------+ | Finished upgrading MetaStore schema from 2.2.0 to 2.3.0 | +---------------------------------------------------------+ 1 row in set, 1 warning (0.00 sec)
注意：不要跳着升级，一步一步来。
8，cloudera管理后台，启动hive，并查看版本
[root@bigserver1 bin]# hive --version WARNING: Use "yarn jar" to launch YARN applications. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hive/lib234/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Hive 2.3.4 Git git://daijymacpro-2.local/Users/daijy/commit/hive -r 56acdd2120b9ce6790185c679223b8b5e884aaf2 Compiled by daijy on Wed Oct 31 14:20:50 PDT 2018 From source with checksum 9f2d17b212f3a05297ac7dd40b65bab0
下篇文章讲一讲，flink-sql，怎么整合hive2.3.4
转载请注明
作者:海底苍鹰
地址:http://blog.51yip.com/hadoop/2391.html
你好，我采用了你【cdh hive 2.1.1 升级到 2.3.4】的方法，通过hive client可以运行，但是我通过beeline就会卡住，任务无法提交的yarn上(我的环境是CDH6.2 hive on spark, spark on yarn，hive2.1.1升级2.3.4)，我看后台异常如下：
Caused by: java.lang.RuntimeException: java.lang.InterruptedException
at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-14.0.1.jar:?]
at org.apache.hive.spark.client.SparkClientImpl.(SparkClientImpl.java:125) ~[hive-exec-2.3.4.jar:2.3.4]
at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80) ~[hive-exec-2.3.4.jar:2.3.4]
at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.createRemoteClient(RemoteHiveSparkClient.java:101) ~[hive-exec-2.3.4.jar:2.3.4]
at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.(RemoteHiveSparkClient.java:97) ~[hive-exec-2.3.4.jar:2.3.4]
at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:73) ~[hive-exec-2.3.4.jar:2.3.4]
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:62) ~[hive-exec-2.3.4.jar:2.3.4]
... 22 more
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method) ~[?:1.8.0_181]
at java.lang.Object.wait(Object.java:502) ~[?:1.8.0_181]
at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:232) ~[netty-all-4.0.52.Final.jar:4.0.52.Final]
at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:34) ~[netty-all-4.0.52.Final.jar:4.0.52.Final]
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:32) ~[netty-all-4.0.52.Final.jar:4.0.52.Final]
at org.apache.hive.spark.client.SparkClientImpl.(SparkClientImpl.java:109) ~[hive-exec-2.3.4.jar:2.3.4]
at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80) ~[hive-exec-2.3.4.jar:2.3.4]
at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.createRemoteClient(RemoteHiveSparkClient.java:101) ~[hive-exec-2.3.4.jar:2.3.4]
at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.(RemoteHiveSparkClient.java:97) ~[hive-exec-2.3.4.jar:2.3.4]
at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:73) ~[hive-exec-2.3.4.jar:2.3.4]
at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:62) ~[hive-exec-2.3.4.jar:2.3.4]
... 22 more
--请问你有遇上个这个问题吗？
hive on spark, spark on yarn，spark目录下hive包是否已更改。spark-hive_2.11-2.4.0-cdh6.3.1.jar
之前没有更改！现在hive-exec-2.3.4.jar拷进去会报别的错误：org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: xxx_table
另外还有一个错误：set hive.plan.serialization.format=javaXML;
Error while processing statement: hive configuration hive.plan.serialization.format does not exists
像这样的该如何解决，谢谢！
是要把spark-hive_2.11-2.4.0-cdh6.3.1.jar换成其他的spark-hive-*.jar?
----------------------------------------
hive on spark, spark on yarn，spark目录下hive包是否已更改。spark-hive_2.11-2.4.0-cdh6.3.1.jar
升级后，sqopp通过hcatalog导数会报错，请问大佬有碰到过吗？