随着数据量的不断增加，又不想增加开发成本，我们选择了tidb数据库，在olap和oltp方面表现都非常的抢眼。为了数据分层，更加方便的提供数据，还需要一个在olap方面有优秀表现的数据存储工具，es我们也在用，不过不能跨索引查询，这会导致单索引会越来越大，不好维护。而clickhouse在olap方面表现优异，并且支持联合查询。
一，zookeeper安装配置
1，安装java
yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel
2，下载zookeeper
wget https://downloads.apache.org/zookeeper/stable/apache-zookeeper-3.6.3-bin.tar.gz tar zxvf apache-zookeeper-3.6.3-bin.tar.gz -C /home/zookeeper
3，配置zookeeper
mkdir -p /home/zookeeper/{data,logs}
# cat /home/zookeeper/conf/zoo.cfg
ticketTime=2000
clientPort=2181
dataDir=/home/zookeeper/data
dataLogDir=/home/zookeeper/logs
initLimit=10
syncLimit=5
server.1=10.0.55.17:2888:3888
server.2=10.0.10.23:2888:3888
server.3=10.0.55.16:2888:3888
server.4=10.0.10.24:2888:3888
echo 1 > /home/zookeeper/data/myid #10.0.55.17机器上
echo 2 > /home/zookeeper/data/myid #10.0.10.23机器上
echo 3 > /home/zookeeper/data/myid #10.0.55.16机器上
echo 4 > /home/zookeeper/data/myid #10.0.10.24机器上
4，配置自启动脚本
# cat /usr/lib/systemd/system/zookeeper.service [Unit] Description=zookeeper After=syslog.target network.target [Service] Type=forking User=root Group=root ExecStart=/home/zookeeper/bin/zkServer.sh start ExecStop=/home/zookeeper/bin/zkServer.sh stop Restart=on-failure [Install] WantedBy=multi-user.target
5，启动zookeeper
# systemctl start zookeeper && systemctl enable zookeeper # jps 8459 QuorumPeerMain //有该进程说明启动成功 13131 Jps
二，clickhouse安装配置
1，安装clickhouse
curl -s https://packagecloud.io/install/repositories/altinity/clickhouse/script.rpm.sh | sudo bash yum install -y clickhouse-server-common.x86_64 clickhouse-server.x86_64 clickhouse-client.x86_64
2，创建目录
mkdir -p /home/clickhouse/tmp /home/clickhouse/logs /home/clickhouse/lock/ chown clickhouse.clickhouse -R /etc/clickhouse-server chown clickhouse.clickhouse -R /home/clickhouse
3，配置config.xml
# cat /etc/clickhouse-server/config.xml
<?xml version="1.0"?>
<yandex>
<!--日志-->
<logger>
<level>error</level>
<log>/home/clickhouse/logs/clickhouse.log</log>
<errorlog>/home/clickhouse/logs/error.log</errorlog>
<size>500M</size>
<count>5</count>
</logger>
<!--本地节点信息-->
<http_port>8123</http_port>
<tcp_port>9000</tcp_port>
<interserver_http_port>9009</interserver_http_port>
<interserver_http_host>clickhouse.test.com</interserver_http_host>
<!--本机域名或IP-->
<!--本地配置-->
<listen_host>0.0.0.0</listen_host>
<max_connections>4096</max_connections>
<keep_alive_timeout>300</keep_alive_timeout>
<max_concurrent_queries>1000</max_concurrent_queries>
<uncompressed_cache_size>8589934592</uncompressed_cache_size>
<mark_cache_size>5368709120</mark_cache_size>
<path>/home/clickhouse/</path>
<tmp_path>/home/clickhouse/tmp/</tmp_path>
<users_config>users.xml</users_config>
<default_profile>default</default_profile>
<default_database>default</default_database>
<!--集群相关配置-->
<remote_servers incl="clickhouse_remote_servers"/>
<zookeeper incl="zookeeper-servers" optional="true"/>
<macros incl="macros" optional="true"/>
<builtin_dictionaries_reload_interval>3600</builtin_dictionaries_reload_interval>
<max_session_timeout>3600</max_session_timeout>
<default_session_timeout>300</default_session_timeout>
<max_table_size_to_drop>0</max_table_size_to_drop>
<merge_tree>
<parts_to_delay_insert>300</parts_to_delay_insert>
<parts_to_throw_insert>600</parts_to_throw_insert>
<max_delay_to_insert>2</max_delay_to_insert>
</merge_tree>
<max_table_size_to_drop>0</max_table_size_to_drop>
<max_partition_size_to_drop>0</max_partition_size_to_drop>
<distributed_ddl>
<!-- Path in ZooKeeper to queue with DDL queries -->
<path>/clickhouse/task_queue/ddl</path>
<cleanup_delay_period>60</cleanup_delay_period>
<task_max_lifetime>86400</task_max_lifetime>
<max_tasks_in_queue>200</max_tasks_in_queue>
</distributed_ddl>
<include_from>/etc/clickhouse-server/metrika.xml</include_from>
</yandex>
4，配置users.xml
# cat /etc/clickhouse-server/users.xml <?xml version="1.0"?> <yandex> <profiles> <default> <max_memory_usage>10000000000</max_memory_usage> <use_uncompressed_cache>0</use_uncompressed_cache> <load_balancing>random</load_balancing> </default> <readonly> <max_memory_usage>10000000000</max_memory_usage> <use_uncompressed_cache>0</use_uncompressed_cache> <load_balancing>random</load_balancing> <readonly>1</readonly> </readonly> </profiles> <quotas> <!-- Name of quota. --> <default> <interval> <queries>0</queries> <errors>0</errors> <result_rows>0</result_rows> <read_rows>0</read_rows> <execution_time>0</execution_time> </interval> </default> </quotas> <users> <default> <!-- PASSWORD=$(base64 < /dev/urandom | head -c8); echo "$PASSWORD"; echo -n "$PASSWORD" | sha256sum | tr -d '-' --> <!-- password UHXQQmhb --> <password>test1234</password> <networks> <ip>::/0</ip> </networks> <profile>default</profile> <quota>default</quota> </default> <ck> <password>test1234</password> <networks> <ip>::/0</ip> </networks> <profile>readonly</profile> <quota>default</quota> </ck> </users> </yandex>
5，配置metrika.xml
# cat /etc/clickhouse-server/metrika.xml <yandex> <!--ck集群节点--> <clickhouse_remote_servers> <clickhouse_test_netjoy> <!--分片1--> <shard> <internal_replication>true</internal_replication> <replica> <host>10.0.55.17</host> <port>9000</port> <user>default</user> <password>test1234</password> </replica> <!--复制集1--> <replica> <host>10.0.10.23</host> <port>9000</port> <user>default</user> <password>test1234</password> </replica> </shard> <!--分片2--> <shard> <internal_replication>true</internal_replication> <replica> <host>10.0.55.16</host> <port>9000</port> <user>default</user> <password>test1234</password> </replica> <!--复制集2--> <replica> <host>10.0.10.24</host> <port>9000</port> <user>default</user> <password>test1234</password> </replica> </shard> </clickhouse_test_netjoy> </clickhouse_remote_servers> <!--zookeeper相关配置--> <zookeeper-servers> <node index="1"> <host>10.0.55.17</host> <port>2181</port> </node> <node index="2"> <host>10.0.10.23</host> <port>2181</port> </node> <node index="3"> <host>10.0.55.16</host> <port>2181</port> </node> <node index="4"> <host>10.0.10.24</host> <port>2181</port> </node> </zookeeper-servers> <macros> <layer>test_netjoy1</layer> <shard>2</shard> <!--分片号--> <replica>10.0.10.23</replica> <!--当前节点IP--> </macros> <networks> <ip>::/0</ip> </networks> <!--压缩相关配置--> <clickhouse_compression> <case> <min_part_size>10000000000</min_part_size> <min_part_size_ratio>0.01</min_part_size_ratio> <method>lz4</method> <!--压缩算法lz4压缩比zstd快, 更占磁盘--> </case> </clickhouse_compression> </yandex>
在这里采用了2个分片，2个副本，macros的配置，同一分片的shard号，一定要一样。
三，启动与集群检查
1，启动
# sudo -u clickhouse clickhouse-server --config-file=/etc/clickhouse-server/config.xml
Processing configuration file '/etc/clickhouse-server/config.xml'.
Including configuration file '/etc/clickhouse-server/metrika.xml'.
Logging error to /home/clickhouse/logs/clickhouse.log
Logging errors to /home/clickhouse/logs/error.log
Logging error to console
2021.08.17 10:37:28.281597 [ 12288 ] {} <Error> Application: DB::Exception: A setting 'receive_timeout' appeared at top level in config /etc/clickhouse-server/config.xml. But it is user-level setting that should be located in users.xml inside <profiles> section for specific profile. You can add it to <profiles><default> if you want to change default value of this setting. You can also disable the check - specify <skip_check_for_incorrect_settings>1</skip_check_for_incorrect_settings> in the main configuration file.
如果有配置错误，可以很好的体现出来。
# /etc/init.d/clickhouse-server start //启动
2，集群测试
# netstat -tpnl |grep click tcp 0 0 0.0.0.0:9009 0.0.0.0:* LISTEN 12418/clickhouse-se tcp 0 0 0.0.0.0:8123 0.0.0.0:* LISTEN 12418/clickhouse-se tcp 0 0 0.0.0.0:9000 0.0.0.0:* LISTEN 12418/clickhouse-se
有些进程说明已启动成功
# clickhouse-client -u default --password test1234 -h127.0.0.1 ClickHouse client version 20.8.3.18. Connecting to 127.0.0.1:9000 as user default. Connected to ClickHouse server version 20.8.3 revision 54438. testjianshow databases; SHOW DATABASES ┌─name───────────────────────────┐ │ _temporary_and_external_tables │ │ default │ │ system │ └────────────────────────────────┘
clickhouse集群安装配置
转载请注明
作者:海底苍鹰
地址:http://blog.51yip.com/clickhouse/2516.html