cdh hive 2.1.1 升级到 2.3.4

张映 发表于 2020-04-02

分类目录: hadoop/spark/scala

标签:, ,

最近在玩flink,想让flink sql借用hive的元数据库。可是呢,flink sql能连hive的版本,到目前为止只有2个,hive 1.2.1和hive 2.3.4。

cdh6.3.1装的hive是2.1.1,所以不升级hive,flink sql根本用不了。

1,备份hive 2.1.1元数据库

# mysqldump -uroot  -ptest metastore > ./metastore.sql

2,查看cdh6.3.1默认的hive版本

[root@bigserver1 lib]# hive --version
WARNING: Use "yarn jar" to launch YARN applications.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive 2.1.1-cdh6.3.1
Subversion file:///container.redhat7/build/cdh/hive/2.1.1-cdh6.3.1/rpm/BUILD/hive-2.1.1-cdh6.3.1 -r e8d55f408b4f9aa2648bc9e34a8f802d53d6aab3
Compiled by jenkins on Thu Sep 26 03:03:11 PDT 2019
From source with checksum 75df142c785df1d0b37a9d40fe16169f

3,下载hive 2.3.4

# wget http://mirror.bit.edu.cn/apache/hive/hive-2.3.4/apache-hive-2.3.4-bin.tar.gz
# tar zxvf apache-hive-2.3.4-bin.tar.gz -C /opt/cloudera/
# mv -r /opt/cloudera/apache-hive-2.3.4-bin /opt/cloudera/hive

注意:进行升级前,在clouderat管理后台,停止掉,根hive相关的应用。

4,复制hive2.3.4包到原来的hive中

# cp -r /opt/cloudera/hive/lib /opt/cloudera/parcels/CDH/lib/hive/lib234

5,修改启动hive文件,调用hive2.3.4 jar包

# vim /opt/cloudera/parcels/CDH/lib/hive/bin/hive

# HIVE_LIB=${HIVE_HOME}/lib   //注释本行,改为
HIVE_LIB=${HIVE_HOME}/lib234

6,同步文件

# cd /opt/cloudera/parcels/CDH/lib/hive

# scp -r lib234 bigserver4:/opt/cloudera/parcels/CDH/lib/hive
# scp -r lib234 bigserver3:/opt/cloudera/parcels/CDH/lib/hive
# scp -r lib234 bigserver2:/opt/cloudera/parcels/CDH/lib/hive
# scp -r lib234 bigserver5:/opt/cloudera/parcels/CDH/lib/hive

# scp bin/hive bigserver4:/opt/cloudera/parcels/CDH/lib/hive/bin
# scp bin/hive bigserver3:/opt/cloudera/parcels/CDH/lib/hive/bin
# scp bin/hive bigserver2:/opt/cloudera/parcels/CDH/lib/hive/bin
# scp bin/hive bigserver5:/opt/cloudera/parcels/CDH/lib/hive/bin

//所有节点创建软连接,并分配权限
# ln -s /opt/cloudera/parcels/CDH/lib/hive/lib234/jline-2.12.jar /opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib/jline-2.12.jar
# chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcels

7,升级hive 2.1.1 元数据库 metastore

mysql> use metastore;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> source /opt/cloudera/hive/scripts/metastore/upgrade/mysql/upgrade-2.1.0-to-2.2.0.mysql.sql;
+------------------------------------------------+
| |
+------------------------------------------------+
| Upgrading MetaStore schema from 2.1.0 to 2.2.0 |
+------------------------------------------------+
1 row in set, 1 warning (0.00 sec)

Query OK, 0 rows affected (0.05 sec)
Records: 0 Duplicates: 0 Warnings: 0

Query OK, 30 rows affected (0.01 sec)
Rows matched: 30 Changed: 30 Warnings: 0

Query OK, 0 rows affected (0.06 sec)
Records: 0 Duplicates: 0 Warnings: 0

Query OK, 0 rows affected (0.02 sec)
Records: 0 Duplicates: 0 Warnings: 0

Query OK, 0 rows affected (0.04 sec)
Records: 0 Duplicates: 0 Warnings: 0

Query OK, 0 rows affected (0.00 sec)
Records: 0 Duplicates: 0 Warnings: 0

Query OK, 191 rows affected (0.04 sec)
Records: 191 Duplicates: 0 Warnings: 0

Query OK, 77 rows affected (0.04 sec)
Records: 77 Duplicates: 0 Warnings: 0

Query OK, 0 rows affected (0.03 sec)
Records: 0 Duplicates: 0 Warnings: 0

Query OK, 0 rows affected (0.01 sec)
Records: 0 Duplicates: 0 Warnings: 0

Query OK, 0 rows affected (0.03 sec)
Records: 0 Duplicates: 0 Warnings: 0

Query OK, 0 rows affected (0.01 sec)
Records: 0 Duplicates: 0 Warnings: 0

Query OK, 0 rows affected (0.00 sec)
Records: 0 Duplicates: 0 Warnings: 0

Query OK, 0 rows affected (0.00 sec)
Records: 0 Duplicates: 0 Warnings: 0

Query OK, 0 rows affected (0.03 sec)
Records: 0 Duplicates: 0 Warnings: 0

Query OK, 0 rows affected (0.01 sec)
Records: 0 Duplicates: 0 Warnings: 0

Query OK, 0 rows affected (0.00 sec)
Records: 0 Duplicates: 0 Warnings: 0

Query OK, 0 rows affected (0.00 sec)
Records: 0 Duplicates: 0 Warnings: 0

Query OK, 0 rows affected (0.01 sec)
Records: 0 Duplicates: 0 Warnings: 0

Query OK, 0 rows affected (0.00 sec)
Records: 0 Duplicates: 0 Warnings: 0

Query OK, 0 rows affected (0.00 sec)
Records: 0 Duplicates: 0 Warnings: 0

Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0

+---------------------------------------------------------+
| |
+---------------------------------------------------------+
| Finished upgrading MetaStore schema from 2.1.0 to 2.2.0 |
+---------------------------------------------------------+
1 row in set, 1 warning (0.00 sec)

mysql> source /opt/cloudera/hive/scripts/metastore/upgrade/mysql/upgrade-2.2.0-to-2.3.0.mysql.sql;
+------------------------------------------------+
| |
+------------------------------------------------+
| Upgrading MetaStore schema from 2.2.0 to 2.3.0 |
+------------------------------------------------+
1 row in set, 1 warning (0.00 sec)

Query OK, 0 rows affected (0.01 sec)
Records: 0 Duplicates: 0 Warnings: 0

Query OK, 1 row affected (0.01 sec)
Rows matched: 1 Changed: 1 Warnings: 0

+---------------------------------------------------------+
| |
+---------------------------------------------------------+
| Finished upgrading MetaStore schema from 2.2.0 to 2.3.0 |
+---------------------------------------------------------+
1 row in set, 1 warning (0.00 sec)

注意:不要跳着升级,一步一步来。

8,cloudera管理后台,启动hive,并查看版本

[root@bigserver1 bin]# hive --version
WARNING: Use "yarn jar" to launch YARN applications.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/hive/lib234/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive 2.3.4
Git git://daijymacpro-2.local/Users/daijy/commit/hive -r 56acdd2120b9ce6790185c679223b8b5e884aaf2
Compiled by daijy on Wed Oct 31 14:20:50 PDT 2018
From source with checksum 9f2d17b212f3a05297ac7dd40b65bab0

下篇文章讲一讲,flink-sql,怎么整合hive2.3.4



转载请注明
作者:海底苍鹰
地址:http://blog.51yip.com/hadoop/2391.html

5 条评论

  1. smarctor 留言

    你好,我采用了你【cdh hive 2.1.1 升级到 2.3.4】的方法,通过hive client可以运行,但是我通过beeline就会卡住,任务无法提交的yarn上(我的环境是CDH6.2 hive on spark, spark on yarn,hive2.1.1升级2.3.4),我看后台异常如下:
    Caused by: java.lang.RuntimeException: java.lang.InterruptedException
    at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-14.0.1.jar:?]
    at org.apache.hive.spark.client.SparkClientImpl.(SparkClientImpl.java:125) ~[hive-exec-2.3.4.jar:2.3.4]
    at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80) ~[hive-exec-2.3.4.jar:2.3.4]
    at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.createRemoteClient(RemoteHiveSparkClient.java:101) ~[hive-exec-2.3.4.jar:2.3.4]
    at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.(RemoteHiveSparkClient.java:97) ~[hive-exec-2.3.4.jar:2.3.4]
    at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:73) ~[hive-exec-2.3.4.jar:2.3.4]
    at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:62) ~[hive-exec-2.3.4.jar:2.3.4]
    ... 22 more
    Caused by: java.lang.InterruptedException
    at java.lang.Object.wait(Native Method) ~[?:1.8.0_181]
    at java.lang.Object.wait(Object.java:502) ~[?:1.8.0_181]
    at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:232) ~[netty-all-4.0.52.Final.jar:4.0.52.Final]
    at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:34) ~[netty-all-4.0.52.Final.jar:4.0.52.Final]
    at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:32) ~[netty-all-4.0.52.Final.jar:4.0.52.Final]
    at org.apache.hive.spark.client.SparkClientImpl.(SparkClientImpl.java:109) ~[hive-exec-2.3.4.jar:2.3.4]
    at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:80) ~[hive-exec-2.3.4.jar:2.3.4]
    at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.createRemoteClient(RemoteHiveSparkClient.java:101) ~[hive-exec-2.3.4.jar:2.3.4]
    at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.(RemoteHiveSparkClient.java:97) ~[hive-exec-2.3.4.jar:2.3.4]
    at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:73) ~[hive-exec-2.3.4.jar:2.3.4]
    at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:62) ~[hive-exec-2.3.4.jar:2.3.4]
    ... 22 more

    --请问你有遇上个这个问题吗?

  2. 张映 留言

    hive on spark, spark on yarn,spark目录下hive包是否已更改。spark-hive_2.11-2.4.0-cdh6.3.1.jar

  3. smarctor 留言

    之前没有更改!现在hive-exec-2.3.4.jar拷进去会报别的错误:org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: xxx_table

    另外还有一个错误:set hive.plan.serialization.format=javaXML;
    Error while processing statement: hive configuration hive.plan.serialization.format does not exists

    像这样的该如何解决,谢谢!

  4. xuanqing 留言

    是要把spark-hive_2.11-2.4.0-cdh6.3.1.jar换成其他的spark-hive-*.jar?
    ----------------------------------------
    hive on spark, spark on yarn,spark目录下hive包是否已更改。spark-hive_2.11-2.4.0-cdh6.3.1.jar

  5. ccc 留言

    升级后,sqopp通过hcatalog导数会报错,请问大佬有碰到过吗?