随着数据量的不断增加,又不想增加开发成本,我们选择了tidb数据库,在olap和oltp方面表现都非常的抢眼。为了数据分层,更加方便的提供数据,还需要一个在olap方面有优秀表现的数据存储工具,es我们也在用,不过不能跨索引查询,这会导致单索引会越来越大,不好维护。而clickhouse在olap方面表现优异,并且支持联合查询。
一,zookeeper安装配置
1,安装java
yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel
2,下载zookeeper
wget https://downloads.apache.org/zookeeper/stable/apache-zookeeper-3.6.3-bin.tar.gz tar zxvf apache-zookeeper-3.6.3-bin.tar.gz -C /home/zookeeper
3,配置zookeeper
mkdir -p /home/zookeeper/{data,logs} # cat /home/zookeeper/conf/zoo.cfg ticketTime=2000 clientPort=2181 dataDir=/home/zookeeper/data dataLogDir=/home/zookeeper/logs initLimit=10 syncLimit=5 server.1=10.0.55.17:2888:3888 server.2=10.0.10.23:2888:3888 server.3=10.0.55.16:2888:3888 server.4=10.0.10.24:2888:3888 echo 1 > /home/zookeeper/data/myid #10.0.55.17机器上 echo 2 > /home/zookeeper/data/myid #10.0.10.23机器上 echo 3 > /home/zookeeper/data/myid #10.0.55.16机器上 echo 4 > /home/zookeeper/data/myid #10.0.10.24机器上
4,配置自启动脚本
# cat /usr/lib/systemd/system/zookeeper.service [Unit] Description=zookeeper After=syslog.target network.target [Service] Type=forking User=root Group=root ExecStart=/home/zookeeper/bin/zkServer.sh start ExecStop=/home/zookeeper/bin/zkServer.sh stop Restart=on-failure [Install] WantedBy=multi-user.target
5,启动zookeeper
# systemctl start zookeeper && systemctl enable zookeeper # jps 8459 QuorumPeerMain //有该进程说明启动成功 13131 Jps
二,clickhouse安装配置
1,安装clickhouse
curl -s https://packagecloud.io/install/repositories/altinity/clickhouse/script.rpm.sh | sudo bash yum install -y clickhouse-server-common.x86_64 clickhouse-server.x86_64 clickhouse-client.x86_64
2,创建目录
mkdir -p /home/clickhouse/tmp /home/clickhouse/logs /home/clickhouse/lock/ chown clickhouse.clickhouse -R /etc/clickhouse-server chown clickhouse.clickhouse -R /home/clickhouse
3,配置config.xml
# cat /etc/clickhouse-server/config.xml <?xml version="1.0"?> <yandex> <!--日志--> <logger> <level>error</level> <log>/home/clickhouse/logs/clickhouse.log</log> <errorlog>/home/clickhouse/logs/error.log</errorlog> <size>500M</size> <count>5</count> </logger> <!--本地节点信息--> <http_port>8123</http_port> <tcp_port>9000</tcp_port> <interserver_http_port>9009</interserver_http_port> <interserver_http_host>clickhouse.test.com</interserver_http_host> <!--本机域名或IP--> <!--本地配置--> <listen_host>0.0.0.0</listen_host> <max_connections>4096</max_connections> <keep_alive_timeout>300</keep_alive_timeout> <max_concurrent_queries>1000</max_concurrent_queries> <uncompressed_cache_size>8589934592</uncompressed_cache_size> <mark_cache_size>5368709120</mark_cache_size> <path>/home/clickhouse/</path> <tmp_path>/home/clickhouse/tmp/</tmp_path> <users_config>users.xml</users_config> <default_profile>default</default_profile> <default_database>default</default_database> <!--集群相关配置--> <remote_servers incl="clickhouse_remote_servers"/> <zookeeper incl="zookeeper-servers" optional="true"/> <macros incl="macros" optional="true"/> <builtin_dictionaries_reload_interval>3600</builtin_dictionaries_reload_interval> <max_session_timeout>3600</max_session_timeout> <default_session_timeout>300</default_session_timeout> <max_table_size_to_drop>0</max_table_size_to_drop> <merge_tree> <parts_to_delay_insert>300</parts_to_delay_insert> <parts_to_throw_insert>600</parts_to_throw_insert> <max_delay_to_insert>2</max_delay_to_insert> </merge_tree> <max_table_size_to_drop>0</max_table_size_to_drop> <max_partition_size_to_drop>0</max_partition_size_to_drop> <distributed_ddl> <!-- Path in ZooKeeper to queue with DDL queries --> <path>/clickhouse/task_queue/ddl</path> <cleanup_delay_period>60</cleanup_delay_period> <task_max_lifetime>86400</task_max_lifetime> <max_tasks_in_queue>200</max_tasks_in_queue> </distributed_ddl> <include_from>/etc/clickhouse-server/metrika.xml</include_from> </yandex>
4,配置users.xml
# cat /etc/clickhouse-server/users.xml <?xml version="1.0"?> <yandex> <profiles> <default> <max_memory_usage>10000000000</max_memory_usage> <use_uncompressed_cache>0</use_uncompressed_cache> <load_balancing>random</load_balancing> </default> <readonly> <max_memory_usage>10000000000</max_memory_usage> <use_uncompressed_cache>0</use_uncompressed_cache> <load_balancing>random</load_balancing> <readonly>1</readonly> </readonly> </profiles> <quotas> <!-- Name of quota. --> <default> <interval> <queries>0</queries> <errors>0</errors> <result_rows>0</result_rows> <read_rows>0</read_rows> <execution_time>0</execution_time> </interval> </default> </quotas> <users> <default> <!-- PASSWORD=$(base64 < /dev/urandom | head -c8); echo "$PASSWORD"; echo -n "$PASSWORD" | sha256sum | tr -d '-' --> <!-- password UHXQQmhb --> <password>test1234</password> <networks> <ip>::/0</ip> </networks> <profile>default</profile> <quota>default</quota> </default> <ck> <password>test1234</password> <networks> <ip>::/0</ip> </networks> <profile>readonly</profile> <quota>default</quota> </ck> </users> </yandex>
5,配置metrika.xml
# cat /etc/clickhouse-server/metrika.xml <yandex> <!--ck集群节点--> <clickhouse_remote_servers> <clickhouse_test_netjoy> <!--分片1--> <shard> <internal_replication>true</internal_replication> <replica> <host>10.0.55.17</host> <port>9000</port> <user>default</user> <password>test1234</password> </replica> <!--复制集1--> <replica> <host>10.0.10.23</host> <port>9000</port> <user>default</user> <password>test1234</password> </replica> </shard> <!--分片2--> <shard> <internal_replication>true</internal_replication> <replica> <host>10.0.55.16</host> <port>9000</port> <user>default</user> <password>test1234</password> </replica> <!--复制集2--> <replica> <host>10.0.10.24</host> <port>9000</port> <user>default</user> <password>test1234</password> </replica> </shard> </clickhouse_test_netjoy> </clickhouse_remote_servers> <!--zookeeper相关配置--> <zookeeper-servers> <node index="1"> <host>10.0.55.17</host> <port>2181</port> </node> <node index="2"> <host>10.0.10.23</host> <port>2181</port> </node> <node index="3"> <host>10.0.55.16</host> <port>2181</port> </node> <node index="4"> <host>10.0.10.24</host> <port>2181</port> </node> </zookeeper-servers> <macros> <layer>test_netjoy1</layer> <shard>2</shard> <!--分片号--> <replica>10.0.10.23</replica> <!--当前节点IP--> </macros> <networks> <ip>::/0</ip> </networks> <!--压缩相关配置--> <clickhouse_compression> <case> <min_part_size>10000000000</min_part_size> <min_part_size_ratio>0.01</min_part_size_ratio> <method>lz4</method> <!--压缩算法lz4压缩比zstd快, 更占磁盘--> </case> </clickhouse_compression> </yandex>
在这里采用了2个分片,2个副本,macros的配置,同一分片的shard号,一定要一样。
三,启动与集群检查
1,启动
# sudo -u clickhouse clickhouse-server --config-file=/etc/clickhouse-server/config.xml Processing configuration file '/etc/clickhouse-server/config.xml'. Including configuration file '/etc/clickhouse-server/metrika.xml'. Logging error to /home/clickhouse/logs/clickhouse.log Logging errors to /home/clickhouse/logs/error.log Logging error to console 2021.08.17 10:37:28.281597 [ 12288 ] {} <Error> Application: DB::Exception: A setting 'receive_timeout' appeared at top level in config /etc/clickhouse-server/config.xml. But it is user-level setting that should be located in users.xml inside <profiles> section for specific profile. You can add it to <profiles><default> if you want to change default value of this setting. You can also disable the check - specify <skip_check_for_incorrect_settings>1</skip_check_for_incorrect_settings> in the main configuration file.
如果有配置错误,可以很好的体现出来。
# /etc/init.d/clickhouse-server start //启动
2,集群测试
# netstat -tpnl |grep click tcp 0 0 0.0.0.0:9009 0.0.0.0:* LISTEN 12418/clickhouse-se tcp 0 0 0.0.0.0:8123 0.0.0.0:* LISTEN 12418/clickhouse-se tcp 0 0 0.0.0.0:9000 0.0.0.0:* LISTEN 12418/clickhouse-se
有些进程说明已启动成功
# clickhouse-client -u default --password test1234 -h127.0.0.1 ClickHouse client version 20.8.3.18. Connecting to 127.0.0.1:9000 as user default. Connected to ClickHouse server version 20.8.3 revision 54438. testjian show databases; SHOW DATABASES ┌─name───────────────────────────┐ │ _temporary_and_external_tables │ │ default │ │ system │ └────────────────────────────────┘
转载请注明
作者:海底苍鹰
地址:http://blog.51yip.com/clickhouse/2516.html