spark sql derby 替换成 mysql

张映 发表于 2019-01-09

分类目录: hadoop/spark/scala

标签:, , ,

hive和spark sql 默认用的都是derby,上篇文章说了,hive怎么使用mysql做为元数据库

spark sql使用hive的配置来连接mysql,配置下来挺简单的,并且不需要启动hive

1,所有spark节点,创建mysql连接java包

# ln -s /usr/share/java/mysql-connector-java.jar /bigdata/spark/jars/

2,复制hive主机点的配置hive-site.xml,到所有spark节点,#SPARK_HOME/conf目录下,不用做任何改变。

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>

 <property>
 <name>javax.jdo.option.ConnectionURL</name>
 <value>jdbc:mysql://10.0.0.237:3306/hive?createDatabaseIfNotExist=true</value>
 </property>

 <property>
 <name>javax.jdo.option.ConnectionDriverName</name>
 <value>com.mysql.jdbc.Driver</value>
 </property>

 <property>
 <name>javax.jdo.option.ConnectionUserName</name>
 <value>hive</value>
 </property>

 <property>
 <name>javax.jdo.option.ConnectionPassword</name>
 <value>Netjoy_123</value>
 </property>

 <property>
 <name>hive.exec.scratchdir</name>
 <value>/user/hive/tmp</value>
 </property>

 <property>
 <name>hive.metastore.warehouse.dir</name>
 <value>/user/hive/warehouse</value>
 </property>

 <property>
 <name>hive.querylog.location</name>
 <value>/user/hive/log</value>
 </property>

 <property>
 <name>hive.metastore.schema.verification</name>
 <value>false</value>
 </property>

</configuration>

如果启动hive报

MetaException(message:Hive Schema version 2.3.0 does not match metastore's schema version 1.2.0 Metastore is not upgraded or corrupt)

解决办法:

在所有hive节点,#HIVE_HOME/conf/hive-site.xml添加以下内容

<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
</property>

3,在hive创建数据库和表,然后通过spark-sql去查就知道是不是用了mysql了。



转载请注明
作者:海底苍鹰
地址:http://blog.51yip.com/hadoop/2036.html